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Abstract 

Leading  Indicators  (LI)  were  introduced  to  the  Systems  Engineering  (SE) 
community  in  2007.  These  measures  are  used  to  evaluate  the  effectiveness  of  how  a 
specific  work  activity  is  applied  on  a  project  in  a  manner  that  provides  information  about 
impacts  that  are  likely  to  affect  the  system  performance.  The  Lis  are  designed  to  give  a 
project  manager/systems  engineer  insight  into  where  their  development  project  is  heading 
and  a  chance  to  implement  corrective  actions  early.  This  research  strives  to  apply  Lis  to 
the  testing  community,  specifically  high  speed  sled  testing,  to  improve  the  testing  process 
and,  in  turn,  improve  the  quality  of  the  tests  conducted.  The  thesis  captures  which  SE 
processes  are  emphasized,  valued  and  used  in  the  high  speed  sled  test  community,  then 
identifies  LI  trends  that  are  most  relevant  to  the  high  speed  sled  test  community.  Lastly, 
two  of  the  top  Lis  -  requirements  maturity  and  requirements  validation  -  were  chosen  for 
further  trend  analysis.  Both  of  the  LI  trends  were  broken  down  into  their  suggested 
derived  measures  and  current  project  trends  were  compared  to  historical  trends. 
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LEADING  INDICATOR  ANALYSIS  FOR  HIGH  SPEED  SLED  TEST 

PROGRAMS 


I.  Introduction 


Background 

Multiple  high  speed  sled  test  tracks  around  the  world,  such  as  the  Holloman  High 
Speed  Test  Track  (HHSTT),  have  provided  the  testing  community  with  a  unique  way  to 
evaluate  systems  that  will  be  subjected  to  a  high  speed  flight  environment.  On  average 
three  sled  tests  can  provide  90  percent  of  the  information  at  10  percent  of  the  cost 
compared  to  one  flight  test  (9).  The  systems  that  have  been  tested  range  from  penetrator 
weapons  to  ejection  seats  to  high-speed  rain  erosion  materials.  This  thesis  will 
concentrate  on  the  penetrator  weapon  system  tests,  which  consist  of  accelerating  a  test 
article  (Figure  1)  to  a  desired  speed  and  impacting  it  into  a  target  complex  (Figure  2). 
This  type  of  test  is  also  referred  to  as  an  impact  test. 


Figure  1:  HHSTT  Sled  Train 
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Figure  2:  HHSTT  Target  Complex 


Testing  at  a  high  speed  sled  test  track  is  very  unique.  Sled  tests  are  one  shot  tests; 
no  test  is  identical  to  another.  The  typical  length  of  a  sled  test  project,  from  the  first 
requirement  to  the  project  closeout,  can  range  from  5  to  12  months.  There  is  very  little 
room  for  last  minute  changes,  and  the  process  seeks  to  remove  timely  backtracking 
throughout  the  engineering  activity.  Once  a  test  has  been  launched  the  entire  planning 
process  starts  over. 

Testing  these  systems  on  a  high  speed  sled  track  helps  confirm  performance 

models,  reduce  cost  for  late  design  changes,  identify  and  decrease  safety  risks,  and 

decrease  cost  as  part  of  the  system  development.  High  speed  sled  tests  have  strong 

similarities  to  acquisition  programs  with  test  requirements,  test  infrastructure  design  and 

development,  and  test  planning  and  execution.  Since  each  test  involves  high  risks,  large 

costs  and  a  demanding  schedule,  SE  should  have  a  strong  role.  SE  is  defined  as: 

an  interdisciplinary  approach  and  means  to  enable  the  realization  of 
successful  systems.  It  focuses  on  defining  customer  needs  and  required 
functionality  early  in  the  development  cycle,  documenting  requirements, 
and  then  proceeding  with  design  synthesis  and  system  validation  while 
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considering  the  complete  problem:  operations,  cost  and  schedule, 
performance,  training  and  support,  test,  manufacturing,  and  disposal.  SE 
considers  both  the  business  and  the  technical  needs  of  all  customers  with 
the  goal  of  providing  a  quality  product  that  meets  the  user  needs.  (10) 

A  fairly  new  SE  tool  is  the  SE  leading  indicator  (LI).  An  LI  is  a  tool  used  to  help  predict 

the  outcome  of  a  project  within  a  given  confidence  and  time  range,  “provide  engineering 

leaders  with  the  information  they  need  to  make  informed  decisions  and,  where  necessary, 

take  preventative  or  corrective  action  during  the  program  in  a  proactive  manner.”  (12)  SE 

and  the  LI  tool  play  an  important  role  in  decreasing  risk,  cost,  and  schedule.  (12,13) 

Problem  Statement 

“Systems  engineering  is  widely  used,  but  at  a  relatively  low  level.  ..”(18)  was 
reported  in  a  survey  conducted  on  systems  engineering  in  aerospace  and  defense 
industries.  These  findings  do  not  differ  much  in  the  test  community.  Portions  of  the  SE 
processes  are  used  but  at  relatively  low  levels.  In  some  cases,  such  as  at  the  HHSTT,  the 
test  community  does  not  actively  practice  SE,  yet  unknowingly  uses  some  of  the  SE 
processes  in  their  day  to  day  work.  One  of  the  reasons  SE  is  being  used  at  relatively  low 
levels  is  the  lack  of  confidence  in  the  SE  process.  (18)  Not  all  of  the  Defense  Acquisition 
Guide ’s  (DAG)  16  SE  processes  (3)  or  the  International  Council  on  Systems 
Engineering’s  (INCOSE)  18  SE  processes  (10)  are  utilized  in  the  testing  environment. 
When  a  customer  tests  their  system  or  payload  at  a  high  speed  sled  track  they  are 
interested  in  conducting  a  test  that  satisfies  requirements,  is  low  risk,  on  schedule,  and 
within  budget.  Applying  the  use  of  LI  tools  to  a  high  speed  sled  test  may  help  to  improve 
the  sled  test  process  and  in  turn  improve  the  quality  of  the  tests  conducted. 
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Research  Focus 


The  focus  of  this  research  effort  was  to  evaluate  the  potential  use  of  Lis  on  a  high¬ 
speed  sled  test  during  impact  sled  test  projects.  During  this  research,  the  following 
questions  were  answered  by  conducting  a  quantitative  study  of  past  and  current  impact 
sled  tests: 

•  What  SE  activities  does  the  high  speed  sled  track  community  currently 
emphasize,  value  and  use? 

•  Which  of  the  1 8  LI  trends  are  most  relevant  to  a  high  speed  sled  track 
environment? 

•  How  do  the  current  project  trend  lines  compare  to  the  historical  trend  lines 
for  different  LI  trends? 


Methodology 

The  methodology  of  this  thesis  is  focused  on  the  application  of  the  SE  processes 
through  the  use  of  Lis  during  the  entire  life  span  of  an  impact  sled  test  project  which 
includes  the  early  planning  phase  from  the  point  the  Project  Manager  (PM)  receives  the 
project.  The  methodology  of  this  thesis  also  focuses  on  how  Lis  can  help  improve 
portions  of  an  impact  sled  test  project.  The  first  step  is  to  determine  what  the  top  SE 
processes  and  LI  trends  are  for  the  high  speed  test  track  environment  and  to  chose  a  few 
LI  trends  on  which  to  collect  historical  and  current  projects  data.  The  second  step  is  to 
determine  the  historical  trend  line  for  each  of  the  chosen  LI  trends.  The  third  step  is  to 
obtain  trend  lines  for  current  projects  for  each  of  the  chosen  LI  trends.  The  last  step  is  to 
compare  the  historical  trend  lines  to  the  current  trends  lines.  By  comparing  the  trend 
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lines  it  can  be  determined  if  the  current  trend  lines  follow  the  historical  trend  lines  and  if 


Lis  are  a  valuable  resource  for  a  high  speed  sled  test  project  manager/engineer. 

Assumptions/Limitations 

An  assumption  for  this  thesis  is  the  use  of  LI  trends  is  still  appropriate  for  short 
unique  sled  tests.  The  historical  trend  lines  created  in  this  thesis  are  for  HHSTT  impact 
tests.  Other  types  of  sled  tests  and  other  sled  test  facilities  have  different  historical 
trends.  Also,  the  PM  will  need  to  be  very  familiar  with  their  test  to  determine  if  their 
project  Li’s  should  follow  the  historic  trends. 

Implications 

If  successful,  the  results  of  this  thesis  give  an  insight  into  the  use  of  LI  trends  in  a 
high  speed  sled  test  environment  and  show  that  they  can  be  useful.  A  secondary  result 
promotes  the  use  of  LI  trends  and  SE  processes  in  the  test  environment. 

Preview 

Chapter  II  presents  the  literature  review,  which  includes  research  on  SE  in  the 
testing  community,  the  SE  processes  and  Lis.  Chapter  III  covers  the  methodology  used 
to  determine  which  SE  processes  the  high  speed  sled  test  community  currently 
emphasizes,  values  and  uses;  what  LI  trends  are  most  relevant  for  use  in  a  sled  track 
environment  and  should  be  used;  past  historical  data;  and  how  the  chosen  historical  LI 
trend  lines  compare  to  current  LI  trend  lines.  Chapter  IV  presents  a  summary  of  the 
findings.  Chapter  V  concludes  the  thesis,  provides  discussion  of  the  results  and  discusses 
future  uses  of  Lis. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  address  the  resources  used  during  the 
information  gathering  phase.  This  chapter  will  supply  background  on  SE  in  the  testing 
community,  SE  processes,  and  LI  trends. 

Systems  Engineering  (SE)  in  the  Testing  Community 

Interest  in  applying  SE  to  a  test  environment  is  present  in  the  SE  community  as 
evidenced  by  a  Systems  Engineering  and  Test  and  Evaluation  (T&E)  conference  held 
yearly  where  various  SE  and  T&E  organizations  come  together  to  discuss  the  use  of  SE 
during  testing  projects.  (16,17)  SE  and  the  SE  processes  work  best  in  the  T&E 
community  when  a  test  is  treated  as  a  full  program  or  project.  Just  like  any  other 
program,  SE  processes  should  be  used  throughout  the  testing  process  from  inception 
through  completion.  (3,10) 

System  Engineering  (SE)  Processes 

The  INCOSE  Systems  Engineering  Handbook  (10)  and  the  Defense  Acquisition 
Guidebook  (DAG)  (3)  both  contain  information  and  guidance  on  the  SE  processes  and 
how  to  properly  use  them.  The  DAG  is  targeted  toward  the  Department  of  Defense 
(DoD)  and  mainly  used  by  military  and  DoD  contractors.  The  INCOSE  Systems 
Engineering  Handbook  is  targeted  to  and  used  by  the  general  public. 

According  to  the  DAG  there  are  16  key  processes  that  should  be  used  during  a 
program’s  life  cycle  (Figure  3).  They  are  split  into  two  categories,  technical  management 
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processes  and  technical  processes.  The  technical  management  processes  are  used  to 


manage  the  technical  development  of  a  system.  These  processes  are  normally  conducted 


in  increments.  The  technical  processes  are  used  to  design  the  system.  This  also  includes 


the  systems  and  equipment  that  support  the  main  system. 


Figure  3:  DAG  Systems  Engineering  Processes  (2) 

According  to  the  1NCOSE  Systems  Engineering  Handbook  there  are  1 8  different 
processes  split  into  two  different  categories;  project  processes  and  technical  processes 
(Figure  4).  Project  processes  are  similar  to  the  DAG  technical  management  processes. 
Both  books  have  decision,  planning,  assessment,  risk,  and  configuration  processes. 
Systems  Engineering  Handbook ’s  information  management  and  measurement  processes 
combined  are  very  similar  to  DAG’s  technical  data  management  and  can  achieve  the 
same  results.  Technical  processes  for  both  books  are  also  very  similar.  The  Systems 
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Engineering  Handbook  has  all  of  the  DAG  technical  processes  plus  the  addition  of  the 
operation,  maintenance  and  disposal  processes. 


TECHNICAL  PROCESSES 


Stakeholder 

Requirements 

Definition 

Requirements 

Analysis 

Architectural 

Design 

Implementation 


Integration 


Verification 


Transition 


Validation 


Operation 


Disposal 


Figure  4:  INCOSE  Systems  Engineering  Processes  (10) 

Both  the  DAG  and  Systems  Engineering  Handbook  offer  insight  into  SE  and  the 
SE  process.  The  decision  on  which  book  to  use  is  dependent  on  the  organization  or 
person  using  the  processes.  The  DAG  is  targeted  towards  military  programs  and  is 
mainly  used  by  Department  of  Defense  (DoD)  organizations  and  DoD  contractors.  The 
Systems  Engineering  Handbook  is  targeted  towards  and  used  more  by  the  general  SE 
population.  Both  books  are  extremely  useful  when  conducting  SE. 


Leading  Indicators  (LI)  Trends 

According  to  the  Systems  Engineering  Leading  Indicator  Guide  “a  leading 
indicator  is  a  measure  for  evaluating  the  effectiveness  of  how  a  specific  activity  is  applied 
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on  a  project  in  a  manner  that  provides  information  about  impacts  that  are  likely  to  affect 
the  system  performance  objectives.”  (12)  An  LI  is  a  tool  used  to  help  predict  the 
outcome  of  a  project  within  a  given  confidence  and  time  range  which  “provide [s] 
engineering  leaders  with  the  information  they  need  to  make  informed  decisions  and, 
where  necessary,  take  preventative  or  corrective  action  during  the  program  in  a  proactive 
manner.”  (13)  In  2007,  the  first  Systems  Engineering  Leading  Indicator  Guide  (version 
1.0)  was  issued  with  these  13  LI  trends. 

1 .  Requirements 

2.  System  Definition  Change  Backlog 

3.  Interface 

4.  Requirements  Validation 

5.  Requirements  Verification 

6.  Work  Product 

7.  Review  Action  Closure 

8.  Risk  Exposure 

9.  Risk  Handling 

10.  Technology  Maturity 

11.  Technical  Measurement 

12.  Systems  Engineering  Staffing  &  Skills 

13.  Process  Compliance 


According  to  version  2.0  of  the  Systems  Engineering  Leading  Indicator  Guide 
released  in  2010  five  additional  LI  trends  were  added. 

14.  Facility  and  Equipment  Availability 

15.  Defect  and  Error 

16.  System  Affordability 

17.  Architecture 

18.  Schedule  and  Cost  Pressure 


Traditional  methods  used  to  determine  the  trend  of  a  program  rely  heavily  on 
historical  data  and  some  current  information.  Although  historical  data  is  used,  Lis  rely 
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heavily  on  the  present  and  look  to  the  future.  “While  leading  indicators  appear  similar  to 
existing  measures  and  often  use  the  same  base  information,  the  difference  lies  in  how  the 
information  is  gathered,  evaluated,  interpreted,  and  used  to  provide  a  forward  looking 
perspective.”  (12)  Lis  are  intended  to  be  used  on  current  and  ongoing  projects  and  use  a 
graphical  presentation  to  convey  the  information. 

Lis  consist  of  three  parts:  characteristics,  conditions,  and  indications.  The 
characteristics  include  needed  information,  leading  insight,  base  measure  specification, 
attributes,  derived  measures,  and  indicators  (Appendix  A).  The  base  measures  are  used 
to  determine  the  trend  and  are  defined  by  a  specific  measurement  method.  The  derived 
measures  are  formed  by  the  base  measures  and  describe  one  or  more  measures.  (12)  For 
example,  one  of  the  system  definition  change  backlog  base  measures  could  be  requests 
for  change.  An  example  of  a  derived  measure  linked  to  this  base  measure  would  be 
approval/closure  rates  which  tracks  the  number  of  changes  requested  versus  the  number 
of  change  requests  approved.  The  condition  is  the  type  of  project  or  system  that  is  being 
tracked.  The  outcome  of  combining  the  characteristics  and  a  condition  is  an  indication. 
The  indication  gives  the  organization  a  predicted  behavior  of  the  project  for  that  specific 
trend.  This  information  is  normally  presented  graphically  (Figure  5). 
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Figure  5:  Sample  LI  Trend  Graphs  (11) 

The  time  between  collecting  data  for  the  chosen  Lis  will  vary  by  the  type  of 
project,  organization,  and  trend.  PMs  may  want  to  collect  data  weekly,  monthly,  or 
quarterly.  Data  for  the  requirements  trends  LI  might  be  collected  every  week  while 
collecting  data  for  the  process  compliance  trends  leading  indicator  might  only  be 
collected  every  month  for  the  same  project. 

For  a  LI  trend  to  be  useful,  the  correct  number  and  type  of  trends  must  be  chosen. 
Choosing  the  wrong  trend  will  give  a  false  outcome.  Choosing  too  many  trends  will  take 
time  away  from  the  project  and  be  too  cumbersome  to  be  useful.  Choosing  too  few  of  the 
trends  will  not  render  sufficient  information  to  make  informed  decisions.  The  number 
and  type  of  LI  trends  to  be  used  is  dependent  upon  the  size  of  the  project,  how  much  time 


11 


the  person  or  organization  has  to  spend  collecting  and  tracking  the  data  and  the  type  of 
project.  The  requirements,  requirements  validation,  requirements  verification  and  system 
definition  change  log  trends  are  useful  for  projects  that  have  a  large  number  of 
requirements,  a  large  number  of  last  minute  requirements  or  a  large  number  of  changes  in 
the  requirements.  The  interface  trend  is  useful  for  projects  that  have  a  system  with 
multiple  parts  or  a  system  that  will  interface  with  multiple  outside  systems.  The  work 
product  approval,  review  action  closure,  system  engineering  staffing  and  skill,  and 
process  compliance  trends  are  all  used  to  track  different  aspects  within  the  organization 
throughout  the  progression  of  a  project.  Risk  exposure  and  risk  handling  trends  deal  with 
program  risk  and  are  useful  for  programs  with  a  large  number  of  different  risks  or  high 
risk  items.  Both  technology  maturity  and  technical  measurement  trends  help  projects  that 
have  a  significant  amount  of  and  new  technology  associated  with  them.  The  facility  and 
equipment  availability  trends  are  useful  for  organizations  that  use  different  facilities  and 
types  of  equipment  on  multiple  projects.  The  defect  and  error  trend  is  useful  with 
software  development.  The  system  affordability  trends  and  schedule  and  cost  pressure 
trends  are  useful  for  projects  that  are  highly  concerned  with  the  budget  and  staying  on 
schedule.  The  architecture  trend  is  useful  for  large  projects. 

All  of  these  Lis  are  useful  and  when  used  properly  will  help  to  make  a  program 
successful.  However,  not  all  Lis  are  useful  for  every  project  and  choosing  inappropriate 
Lis  or  the  wrong  number  of  Lis  can  be  harmful  to  a  project.  Without  insight  into  the 
project,  Lis  generate  useless  graphs  that  can  give  misleading  information.  However,  if 
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the  Systems  Engineer  or  PM  chooses  the  right  combination  of  Lis,  then  the  use  of  Lis  can 
be  very  successful. 

Summary 

The  interest  in  using  SE  in  testing  environments  has  been  presented.  Both  the 
DAG  and  Systems  Engineering  Handbook  give  beneficial  insight  into  the  SE  processes 
and  both  are  useful  for  projects  in  a  testing  environment.  LI  trends  can  also  be  used  in  a 
testing  environment  that  appear  to  be  similar  to  developmental  projects  -  that  is,  the 
projects  have  requirements,  develop  modifications  to  existing  components  ( test  sleds  and 
impact  targets),  plan  to  collect  data,  integrate  non-developed  items  (customer  provided 
test  articles)  and  manage  within  cost  and  schedule. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  illustrate  the  approach  taken  to  determine  which 
SE  processes  the  high  speed  sled  test  community  currently  emphasizes,  values  and  uses, 
and  which  LI  trends  are  most  relevant  to  a  sled  track  environment.  It  will  also  illustrate 
the  approach  taken  for  data  collection  and  compare  the  historical  trend  line  to  a  current 
trend  line  for  different  LI  trends. 

Choosing  System  Engineering  Processes 

Determining  which  SE  processes  the  high  speed  sled  test  community  currently 
emphasizes,  values  and  uses  was  completed  using  two  different  methods.  The  first 
method  was  to  research  SE  processes  that  are  currently  and  prominently  being  used.  The 
second  method  was  to  determine  what  SE  processes  are  thought  to  be  useful  to  the  high 
speed  sled  test  community  whether  or  not  they  are  currently  being  utilized. 

To  obtain  information  on  the  current  SE  processes  that  are  being  prominently 
used  in  a  high  speed  sled  test  environment,  the  HHSTT  operations  and  procedures  were 
reviewed.  This  review  included  project  notes,  squadron  operational  instructions  (SOI) 
and  meeting  minutes.  Personal  experience  conducting  sled  tests  was  also  used  to  obtain 
this  information. 

To  determine  what  SE  processes  are  thought  to  be  useful,  an  SE  processes 
questionnaire  was  discussed  with  subject  matter  experts  (SMEs)  at  the  HHSTT 
(Appendix  B).  Data  collection  was  received  from  ten  members  of  the  squadron  including 
upper  management,  current  and  past  PMs,  data  engineers  and  test  engineers.  The  data  set 
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rated  the  importance  of  each  process  from  1  to  5,  with  1  being  most  useful,  to  the  projects 
conducted  at  the  HHSTT.  The  second  part  of  the  data  set  ranked  the  16  processes  in 
usefulness  to  the  HHSTT  from  1  to  16,  with  1  being  the  most  useful. 

Choosing  Leading  Indicator  Trends 

The  information  obtained  from  determining  the  SE  processes  the  high  speed  sled 
test  community  currently  emphasizes,  values  and  uses,  and  a  second  questionnaire 
(Appendix  C)  were  used  to  determine  which  LI  trends  are  most  relevant  to  the  sled  track 
environment. 

The  SE  processes  can  be  linked  to  different  LI  trends.  The  Systems  Engineering 
Leading  Indicators  Guide  links  the  LI  trends  to  the  INCOSE  SE  processes.  For  this 
thesis,  data  was  collected  from  the  HHSTT,  a  DoD  organization.  The  SE  guide  most 
appropriate  for  the  HHSTT  is  Chapter  4  of  the  Defense  Acquisition  Guide  (DAG).  For 
this  reason  it  is  important  to  link  the  DAG  SE  processes  to  the  different  LI  trends  (Table 
1).  Using  Table  1  the  top  LI  trends  can  be  determined  from  the  top  SE  processes. 

To  obtain  a  complete  picture  of  the  most  relevant  LI  trends,  a  second 
questionnaire  was  discussed  with  the  same  ten  SMEs  from  the  HHSTT  squadron. 
Similarly  to  the  previous  data  set,  each  LI  trend  was  rated  from  1  to  5  with  1  being  most 
useful.  The  second  part  of  the  data  set  ranked  the  18  trends  in  usefulness  from  1  to  18 
with  1  being  the  most  useful.  The  top  most  useful  Lis  were  determined  by  combining 
this  survey  with  the  SE  processes  data. 
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Leading  Indicator  Trends 


Table  1:  SE  Processes  vs.  LI  Trends 


System  Engineering  Processes 

Technical  Planning 

Requirements  Management 

Configuration  Management 

Technical  Assessment 

Decision  Analysis 

Risk  Management 

Interface  Management 

Data  Management 

Stakeholder  Requirements 

Definition 

Requirements  Analysis 

Architecture  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Requirements 

X 

X 

X 

System  Definition 
Change  Backlog 

X 

X 

X 

X 

X 

Interface 

X 

Requirements 

Validation 

X 

X 

X 

Requirements 

Verification 

X 

X 

Work  Product 
Approval 

X 

X 

Review  Action 

Closure 

X 

X 

Risk  Exposure 

X 

Risk  Treatment 

X 

Technology 

Maturity 

X 

X 

X 

X 

Technical 

Measurement 

X 

X 

X 

X 

X 

X 

Systems 
Engineering 
Staffing  &  Skill 

X 

X 

Process 

Compliance 

X 

Facility  and 
Equipment 
Availability 

X 

X 

Defect/Error 

X 

X 

System 

Affordability 

X 

X 

X 

X 

Architecture 

X 

Schedule  and  Cost 

Pressure 

X 

X 
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Historical  Trend  Lines 


LI  trend  lines  from  past  impact  tests  needed  to  be  collected.  Two  LI  trends  were 
selected  for  historical  data  collection.  A  historical  trend  line  was  created  from  different 
derived  measures  for  each  of  the  two  chosen  LI  trends. 

Four  past  impact  tests  from  different  customers  were  researched.  Three  separate 
PMs  were  associated  with  these  four  projects.  These  projects  were  selected  because  they 
were  similar  projects,  considered  to  be  typical  impact  tests  for  the  HHSTT,  yet  included 
some  dissimilar  elements.  All  four  tests  were  successful.  Data  was  collected  and 
recorded  in  weekly  increments.  For  trend  lines  over  time,  the  results  were  scaled  to  40 
weeks  (about  10  months)  and  then  an  average  was  taken  for  each  week.  Historical 
bounds  were  used  to  determine  a  valid  trend  line  range  and  used  when  comparing  the 
current  project  trend  lines  to  the  historical  trend  lines.  The  historical  bounds  were  found 
by  determining  the  maximum  and  minimum  values  between  each  of  the  four  historical 
projects  at  each  data  collection  interval.  This  data  was  then  compiled  into  line  graphs. 

Current  Trend  Lines 

A  Microsoft  Excel™  LI  tool  (Appendix  D)  was  created  to  collect  and  record  the 
current  trend  lines  for  two  of  the  top  most  useful  LI  trends.  Users  enter  their  current 
project  start  and  end  date  and  data  collection  rate.  The  tool  then  scales  the  historical 
trend  line  graphs  to  match  the  length  of  the  current  project.  Users  then  enter  their 
information  at  the  intervals  they  have  chosen.  The  end  results  are  graphs  with  both 
historical  and  current  trend  lines  along  with  the  historical  bounds.  This  tool  was  sent  to 
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two  PMs  at  the  HHSTT  who  each  entered  data  from  their  current  impact  tests  for  a  total 
of  three  tests.  These  projects  were  randomly  chosen  by  the  PMs. 

Project  A  is  50  weeks  long  and  the  data  was  collected  every  two  weeks  up  to 
week  30.  The  PM  tracked  only  the  high  level  requirements,  a  total  of  1 1,  for  this  project. 
Project  B  is  20  weeks  long  and  the  data  was  collected  every  two  weeks  up  to  week  12. 
The  PM  tracked  the  high  and  medium  level  requirements,  a  total  of  26,  for  this  project. 
The  third  project,  Project  C,  is  a  short  project  of  only  14  weeks  and  the  data  was  collected 
every  week  up  to  week  1 1 .  The  PM  tracked  high,  medium  and  low  level  requirements  for 
a  total  of  42.  The  graphs  from  the  Microsoft  Excel™  LI  tool  were  used  to  compare  the 
historical  and  current  trend  lines  for  these  tests. 

Summary 

The  top  SE  processes  were  determined  using  knowledge  of  the  high  speed  test 
environment  and  a  questionnaire.  The  top  LI  trends  were  determined  by  using  the  top  SE 
processes  and  a  second  questionnaire.  Historical  data  was  collected  from  different 
HHSTT  impact  projects,  scaled  and  graphed.  Two  HHSTT  PMs  contributed  data  for 
their  current  impact  tests  which  was  then  graphically  compared  to  the  historical  data.  A 
total  number  of  four  past  impact  tests  and  three  current  impact  tests  were  used  to  obtain 
the  data. 
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IV.  Analysis  and  Results 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  show  the  data  collected  and  results  in 
determining  which  SE  processes  the  high  speed  sled  test  community  currently 
emphasizes,  values  and  uses  and  which  LI  trends  are  most  relevant  to  a  sled  track 
environment.  Two  LI  trends  were  selected  to  analyze  the  historical  and  current  trend. 
The  results  and  comparison  of  these  two  LI  trends  are  also  reported  in  this  chapter.  All 
data  collected  was  obtained  from  the  HHSTT  and  squadron  personnel. 

Top  System  Engineering  Processes 

Researching  the  HHSTT  project  notes,  SOIs  and  various  project  meeting  minutes 
found  the  HHSTT  is  prominently  using  SE  processes  in  their  day-to-day  conduct  without 
focusing  on  SE  or  the  SE  processes.  Out  of  the  16  DAG  SE  processes  the  HHSTT  uses 
six  on  a  regular  basis  (Table  2).  These  processes  include  requirements  management,  risk 
management,  stakeholder  requirements  definition,  validation,  verification  and  interface 
management. 
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Table  2:  SE  Processes  used  at  the  HHSTT 


Technical  Planning 
Requirements  Management 
Decision  Analysis 
Technical  Assessment 
Risk  Management 
Data  Management 
Requirements  Analysis 
Stakeholder  Requirements  Definition 
Validation 
Verification 

€es<iguration  Managcmes4 
Integration 

Interface  Management 
Architecture  Design 
Transition 

_ Implementation _ 


The  first  data  collected  from  HHSTT  squadron  members  was  on  SE  processes  to 
determine  what  processes  are  important  to  the  HHSTT,  even  if  they  do  not  currently  use 
the  process.  Because  a  process  is  not  used  does  not  mean  the  process  should  not  be  used, 
The  first  part  of  the  questionnaire  rated  each  of  the  processes  from  one  to  five.  The 
second  part  of  the  questionnaire  ranked  all  of  the  processes  form  one  to  16.  The  results 
from  the  first  portion  of  this  questionnaire  (Table  3)  show  the  top  two  rate  processes  are 
technical  planning  and  requirements  management  with  averages  of  1.20  and  1.40, 
respectively.  Tied  for  third  with  an  average  of  1.60  are  decision  analysis,  technical 
assessment  and  risk  management  processes.  The  results  from  the  second  portion  of  this 
questionnaire  (Table  4)  show  the  technical  planning  and  requirements  management 
processes  are  tied  for  the  top  ranked  processes  with  an  average  of  3.40.  Risk 
management  and  data  management  are  tied  for  third  with  an  average  of  3.70. 
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Table  3:  SE  Processes  Survey  Individually  Rated 


SE  Processes 

Average 

S  Deviation 

Technical  Planning 

1.20 

0.63 

Requirements  Management 

1.40 

0.70 

Decision  Analysis 

1.60 

0.70 

Technical  Assessment 

1.60 

0.70 

Risk  Management 

1.60 

0.84 

Data  Management 

1.90 

1.10 

Requirements  Analysis 

2.00 

0.82 

Stakeholder  Requirements  Definition 

2.50 

0.53 

Validation 

2.80 

1.69 

Verification 

3.10 

1.52 

Configuration  Management 

3.30 

1.06 

Integration 

3.60 

1.17 

Interface  Management 

3.70 

1.06 

Architecture  Design 

3.90 

0.88 

Transition 

4.10 

1.10 

Implementation 

4.50 

0.53 

Table  4:  SE  Processes  Survey 

Group  Ranked 

SE  Processes 

Average 

S  Deviation 

Technical  Planning 

3.40 

1.17 

Requirements  Management 

3.40 

2.12 

Risk  Management 

3.70 

3.02 

Data  Management 

3.70 

3.30 

Stakeholder  Requirements  Definition 

6.00 

3.37 

Technical  Assessment 

6.40 

2.67 

Requirements  Analysis 

6.60 

2.07 

Decision  Analysis 

7.70 

3.97 

Configuration  Management 

9.60 

2.17 

Validation 

9.70 

4.14 

Interface  Management 

10.00 

2.21 

Verification 

11.90 

3.67 

Architecture  Design 

12.00 

1.83 

Integration 

12.00 

2.00 

Implementation 

14.60 

1.07 

Transition 

15.30 

0.95 
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By  combining  all  the  data  from  the  two  questionnaires  and  SE  processes  the 
HHSTT  currently  utilizes,  the  top  three  SE  processes  the  high  speed  sled  test  community 
currently  emphasizes,  values  and  uses  were  determined.  These  processes  are  the 
requirements  management,  risk  management  and  technical  planning  processes  (Table  5). 
Even  though  technical  planning  is  not  prominently  used  currently  at  the  HHSTT,  it  was 
chosen  as  one  of  the  top  two  because  in  both  sections  of  the  questionnaire  it  was  rated 
and  ranked  as  number  one. 


Table  5:  Top  SE  Processes 


SE  Processes 

Currently 

Used 

Questionnaire 
Part  ITop  3 

Questionnaire 
Part  2  Top  3 

Requirements  Management 

X 

2 

1 

Risk  Management 

X 

3 

3 

Technical  Planning 

1 

1 

Interface  Management 

X 

Stakeholder  Requirements  Definition 

X 

Verification 

X 

Validation 

X 

Decision  Analysis 

3 

Technical  Assessment 

3 

Data  Management 

3 

Configuration  Management 

Requirements  Analysis 

Architecture  Design 

Implementation 

Integration 

Transition 

Top  Leading  Indicators  Trends 

Finding  the  top  three  most  relevant  trends  consists  of  looking  at  the  top  SE 
processes  and  the  top  rated  and  ranked  LI  trends.  Linking  the  SE  processes  to  LI  trends 
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results  in  multiple  possibilities  for  the  top  two  trends.  Fifteen  possible  LI  trends  can  be 
linked  to  the  six  prominently  used  SE  processes  (Table  6).  These  trends  include 
requirements,  system  definition  change  backlog,  requirements  validation,  requirements 
verification,  work  product  approval,  review  action  closure,  risk  exposure,  risk  treatment, 
technical  measurement,  SE  staffing  and  skill,  process  compliance,  facility  and  equipment 
availability,  defect/error,  system  affordability,  and  schedule  and  cost  pressure.  Any  of 
these  fifteen  trends  may  be  useful.  More  information  is  needed  to  narrow  the  selection 
down  to  the  top  three  most  relevant  LI  trends. 

Taking  into  account  the  top  three  SE  processes  found  in  the  above  section, 
requirements  management,  risk  management  and  technical  planning  processes,  the 
number  of  LI  trends  decreases  to  ten  (Table  7).  These  trends  include  requirements,  work 
product  approval,  review  action  closure,  risk  exposure,  risk  treatment,  SE  staffing  and 
skill,  process  compliance,  facility  and  equipment  availability,  defect/error,  and  schedule 
and  cost  pressure.  Again,  any  of  these  ten  LI  trends  could  be  relevant.  To  determine 
which  of  these  ten  LI  trends  are  the  top  trends  or  if  any  of  these  are  the  top  trends,  the 
second  questionnaire  was  needed. 
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Table  6:  Prominently  Used  SE  Processes  vs.  LI  Trends 


System  Engineering  Processes 

Technical  Planning 

Requirements  Management 

Configuration  Management 

Technical  Assessment 

Decision  Analysis 

Risk  Management 

Interface  Management 

Data  Management 

Stakeholder  Requirements 

Definition 

Requirements  Analysis 

Architecture  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Requirements 

X 

X 

X 

System  Definition 
Change  Backlog 

X 

X 

X 

X 

X 

Interface 

X 

Requirements 

Validation 

X 

X 

X 

Requirements 

Verification 

X 

X 

Work  Product 
Approval 

X 

X 

Review  Action 

Closure 

X 

X 

Risk  Exposure 

X 

Risk  Treatment 

X 

Technology 

Maturity 

X 

X 

X 

X 

Technical 

Measurement 

X 

X 

X 

X 

X 

X 

Systems 
Engineering 
Staffing  &  Skill 

X 

X 

Process 

Compliance 

X 

Facility  and 
Equipment 
Availability 

X 

X 

Defect/Error 

X 

X 

System 

Affordability 

X 

X 

X 

X 

Architecture 

X 

Schedule  and  Cost 

Pressure 

X 

X 
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Table  7:  Top  SE  Processes  vs.  LI  Trends 


System  Engineering  Processes 

Technical  Planning 

Requirements  Management 

Configuration  Management 

Technical  Assessment 

Decision  Analysis 

Risk  Management 

Interface  Management 

Data  Management 

Stakeholder  Requirements 

Definition 

Requirements  Analysis 

Architecture  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Requirements 

X 

X 

X 

System  Definition 
Change  Backlog 

X 

X 

X 

X 

X 

Interface 

X 

Requirements 

Validation 

X 

X 

X 

Requirements 

Verification 

X 

X 

Work  Product 
Approval 

X 

X 

Review  Action 

Closure 

X 

X 

Risk  Exposure 

X 

Risk  Treatment 

X 

Technology 

Maturity 

X 

X 

X 

X 

Technical 

Measurement 

X 

X 

X 

X 

X 

X 

Systems 
Engineering 
Staffing  &  Skill 

X 

X 

Process 

Compliance 

X 

Facility  and 
Equipment 
Availability 

X 

X 

Defect/Error 

X 

X 

System 

Affordability 

X 

X 

X 

X 

Architecture 

X 

Schedule  and  Cost 

Pressure 

X 

X 
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Data  collected  from  HHSTT  squadron  members  for  the  second  questionnaire 
composed  of  LI  trends  to  determine  what  trends  are  important  to  the  HHSTT.  The  first 
part  of  the  questionnaire  rated  each  of  the  LI  trends  from  one  to  five  while  the  second 
part  ranked  all  of  the  trends  from  one  to  18.  The  findings  from  the  first  part  of  the 
questionnaire  (Table  8)  show  the  most  relevant  LI  trend  is  the  requirements  trend  with  an 
average  of  1.30.  Tied  for  second,  with  an  average  of  1.50,  are  the  requirements 
validation  trend  and  the  facility  and  equipment  availability  trend.  The  results  from  the 
second  part  of  the  questionnaire  (Table  9)  show  the  two  most  relevant  trends  with  an 
average  2.40  are  the  requirements  and  requirements  validation  trends.  The  results  from 
both  portions  of  the  questionnaire  are  consistent  with  each  other  showing  the  same  top 
three  LI  trends. 

Combining  both  the  LI  trends  questionnaire  and  the  results  from  the  SE  processes, 
the  number  of  relevant  LI  trends  drops  to  two,  requirements  and  facility  and  equipment 
availability  trends  (Table  10).  In  order  to  determine  the  top  three  relevant  LI  trends,  a 
third  trend  was  selected  from  the  LI  questionnaire  results  by  its’  ranking  and  rating.  This 
makes  the  top  three  LI  trends  the  requirements,  facility  and  equipment  availability,  and 
requirements  validation  trends.  Two  of  these  trends  are  linked  to  an  SE  process  that  is 
currently  prominently  being  used  in  the  high  speed  testing  community.  The  requirement 
validation  trend  is  a  trend  that  the  high  speed  sled  track  testing  community  believes 
would  be  relevant  tying  second  in  rating  and  first  in  ranking. 
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Table  8:  LI  Trend  Questionnaire  Individually  Ranked 


LI  Trends 

Average 

S  Deviation 

Requirements 

1.30 

0.48 

Requirements  Validation 

1.50 

0.53 

Facility  and  Equipment  Availability 

1.50 

0.53 

Requirements  Verification 

1.80 

0.92 

Schedule  and  Cost  Pressure 

1.90 

0.99 

System  Affordability 

2.40 

1.26 

Technical  Measurement 

2.50 

1.51 

Risk  Exposure 

2.60 

0.97 

Work  Product  Approval 

2.80 

0.79 

Risk  Handling 

2.80 

0.79 

Review  Action  Closure 

3.00 

1.15 

Process  Compliance 

3.00 

1.15 

Defect  and  Errors 

3.30 

1.34 

System  Engineering  Staffing  &  Skill 

3.40 

1.07 

System  Definition  Change  Log 

3.60 

0.70 

Interface 

3.80 

0.92 

Technology  Maturity 

4.00 

1.15 

Architecture 

4.50 

0.85 

Table  9:  LI  Trend  Questionnaire  Group  Ranked 


LI  Trends 

Average 

S  Deviation 

Requirements 

2.40 

3.10 

Requirements  Validation 

2.40 

1.84 

Facility  and  Equipment  Availability 

3.40 

1.17 

Requirements  Verification 

5.30 

3.50 

System  Affordability 

7.20 

2.66 

Schedule  and  Cost  Pressure 

7.30 

2.83 

Review  Action  Closure 

8.10 

4.28 

Defect  and  Errors 

8.90 

3.90 

Risk  Exposure 

9.10 

1.85 

Risk  Handling 

10.30 

2.06 

System  Engineering  Staffing  &  Skill 

10.40 

4.72 

Work  Product  Approval 

10.90 

2.85 

Process  Compliance 

12.00 

3.56 

Technical  Measurement 

12.30 

4.19 

Technology  Maturity 

14.00 

3.40 

Interface 

14.60 

4.70 

System  Definition  Change  Log 

15.40 

3.06 

Architecture 

16.40 

1.78 
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Trend  Lines 


The  requirements  and  requirements  validation  trends  were  chosen  to  create  historical 
trend  lines.  The  Systems  Engineering  Leading  Indicator  Handbook  gives  nine  suggested 
derived  measures  for  the  requirements  trend.  These  measures  include  percent 
requirements  approved,  percent  requirements  growth,  percent  to-be-determined 
(TBD)/to-be-reviewed  (TBR)  closure  variance  per  plan,  percent  requirements  modified, 
estimated  impact  of  requirements  change,  requirement  defect  profile,  requirement  defect 
density,  requirement  defect  leakage  and  cycle  time  for  requirement  changes.  The  first 
five  of  the  nine  suggested  derived  measures  are  relevant  to  the  HHSTT  and  are  a  good  fit. 
The  HHSTT  does  not  actively  track  defects  making  the  sixth,  seventh,  and  eighth  derived 
measures  not  very  useful.  The  ninth  derived  measure  was  not  chosen  because  the 
HHSTT  does  not  precisely  record  overtime  hours  and  the  PMs  do  not  have  the  means  at 
their  disposal  to  determine  the  start  and  stop  time  of  overtime  hours,  only  the  total 
amount  of  overtime  used  each  week.  The  Systems  Engineering  Leading  Indicator 
Handbook  suggests  two  derived  measures  for  the  requirements  validations  trend: 
requirements  validation  rate  and  percent  requirements  validated.  Both  of  these  suggested 
derived  measures  are  relevant  to  the  HHSTT.  The  first  five  derived  measures  from  the 
requirements  trend  and  both  of  the  derived  measures  from  the  requirements  validation 
trends  will  be  used.  When  collecting  data,  the  type  of  requirements  were  divided  into 
customer  requirements  and  squadron  derived  requirements  to  gain  greater  insight  into  the 
projects,  as  suggested  by  the  Systems  Engineering  Leading  Indicator  Handbook.  The 
next  sections  describe  the  seven  LI  measures  used. 
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Leading  Indicator  Trends 


Table  10:  Top  SE  Processes  vs.  Top  LI  Trends 


System  Engineering  Processes 

Technical  Planning 

Requirements  Management 

Configuration  Management 

Technical  Assessment 

Decision  Analysis 

Risk  Management 

Interface  Management 

Data  Management 

Stakeholder  Requirements 

Definition 

Requirements  Analysis 

Architecture  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Requirements 

X 

X 

X 

System  Definition 
Change  Backlog 

X 

X 

X 

X 

X 

Interface 

X 

Requirements 

Validation 

X 

X 

X 

Requirements 

Verification 

X 

X 

Work  Product 
Approval 

X 

X 

Review  Action 

Closure 

X 

X 

Risk  Exposure 

X 

Risk  Treatment 

X 

Technology 

Maturity 

X 

X 

X 

X 

Technical 

Measurement 

X 

X 

X 

X 

X 

X 

Systems 
Engineering 
Staffing  &  Skill 

X 

X 

Process 

Compliance 

X 

Facility  and 
Equipment 
Availability 

X 

X 

Defect/Error 

X 

X 

System 

Affordability 

X 

X 

X 

X 

Architecture 

X 

Schedule  and  Cost 

Pressure 

X 

X 
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Percent  Requirements  Approved  Derived  Measure 

The  trend  lines  for  the  percent  requirements  approved  derived  measure  are 
percentages  graphed  over  time.  This  derived  measure  takes  into  account  the  number  of 
approved  requirements  versus  the  number  of  identified  requirements  and  uses  the 
equation: 

requirements  approved 
requirements  identified 

The  historical  trend  lines  suggest  that  during  the  first  week  of  the  project  between 
50  and  60  percent  of  the  customer  requirements  and  between  30  and  40  percent  of  the 
derived  requirements  should  be  approved  (Figures  6  &  7).  By  the  time  the  project  is  50 
percent  complete  the  trend  lines  suggest  the  majority  of  both  customer  and  derived 
requirements  should  be  approved.  By  the  time  the  project  is  75  percent  complete  all  the 
requirements  should  be  approved.  At  no  point  do  any  of  the  historical  bounds  differ  from 
the  trend  line  more  than  32  percent  for  the  customer  requirements  and  3 1  percent  for  the 
derived  requirements.  Having  the  project  trend  lines  above  the  historical  trend  lines  is  a 
positive  result  and  indicates  a  stable  project.  Having  the  project  trend  lines  below  the 
historical  trend  lines  is  a  negative  result.  If  the  project  trend  line  is  significantly  below 
the  historical  trend  line,  the  PM  should  investigate  why  requirements  are  not  being 
approved  and  determine  if  actions  need  to  be  taken  to  ensure  the  requirements  are  being 
approved  in  a  timely  manner. 
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Figure  6:  Percent  of  Customer  Requirements  Approved  Historical  Trend  Lines 


Figure  7:  Percent  of  Derived  Requirements  Approved  Historical  Trend  Lines 

Combining  the  project  trend  lines  with  the  historical  trend  lines  gives  insight  into 
where  the  project  is  heading  and  if  there  need  to  be  any  corrections  made.  Project  A’s 
customer  requirements  trend  line  falls  within  the  historical  bounds  50  percent  of  the  time 
while  the  derived  requirements  trend  lines  falls  within  the  bounds  57  percent  of  the  time 
(Figures  8  &  9).  The  trend  lines  suggest  the  customer  requirements  have  all  been 
approved  and  are  well  ahead  of  the  curve.  At  no  point  during  the  project  does  the 
customer  requirements  trend  line  fall  below  the  lower  historical  bound,  and  by  week  nine, 
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the  trend  line  shoots  well  above  the  upper  bound.  In  this  case  not  following  the  historical 
trend  line  is  a  positive  result  because  the  number  of  requirements  approved  is  ahead  of 
historical  norms.  The  project  derived  requirements  trend  line  shows  that  the  rate  at  which 
requirements  are  being  approved  is  slower  than  the  customer  requirements,  even  though 
the  trend  line  is  within  the  historical  bounds  by  7  percent  more  than  the  customer 


requirements  trend  line. 


Figure  8:  Project  A  Percent  of  Customer  Requirements  Approved  Trend  Lines 
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Figure  9:  Project  A  Percent  of  Derived  Requirements  Approved  Trend  Lines 

Project  B’s  trend  lines  follow  the  historical  trend  lines  more  closely  and  fall 
within  the  historical  bounds  for  the  customer  requirements  and  derived  requirements  67 
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percent  and  50  percent  of  the  time,  respectively  (Figures  10  &  11).  The  project’s 
customer  requirements  trend  line  follows  the  historical  trend  line  closely  until  about  week 
eight  when  the  historical  trend  starts  to  increase,  leaving  the  project  trend  line  behind. 
This  should  raise  a  red  flag  for  PM  and  should  be  investigated  due  to  the  fact  that  the 
project  trend  line  does  not  only  fall  below,  with  a  maximum  difference  of  17  percent,  the 
historical  trend  line,  but  it  also  falls  below  the  historical  bound  line.  By  week  12,  the 
customer  project  trend  line  still  has  not  caught  up  to  the  historical  trend  line  and  barely 
skims  the  historical  bound  line  by  one  percent.  This  leads  to  the  conclusion  that  there 
might  be  a  problem.  If  not  handled  soon,  the  schedule  might  slip  or  the  cost  might 
increase.  For  the  majority  of  the  project,  the  derived  requirements  project  trend  line  is 
above  or  very  close  to  the  historical  line  and  never  falls  below  the  historical  bound  line. 
This  should  not  be  a  concern  for  the  PM. 


Customer 
Requirements 
Historical  Trend  Line 

. Customer 

Requirements 
Historical  Bounds 
=  =  Customer 

Requirements  Project 
Trend  Line 

- Current  Week 


Figure  10:  Project  B  Percent  of  Customer  Requirements  Approved  Trend  Lines 
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Figure  11:  Project  B  Percent  of  Derived  Requirements  Approved  Trend  Lines 

Project  C’s  trend  lines  are  very  different  from  the  historical  trend  lines  and 
generally  do  not  fall  within  the  historic  bounds  with  only  45  percent  of  the  customer 
requirements  trend  line  and  0  percent  of  the  derived  requirements  trend  line  within  the 
bounds  (Figures  12  &  13).  According  to  the  historical  trend  lines  all  requirements  should 
be  approved  by  week  10.  The  project  trend  lines  show  that  about  80  percent  of  the 
customer  requirements  and  55  percent  of  the  derived  requirements  have  been  approved. 
Without  the  historical  trend  lines,  the  PM  may  view  these  results  as  normal  because  for 
most  projects  it  may  take  a  few  weeks  for  the  requirements  to  be  approved.  However, 
with  the  historical  trend  lines  scaled  to  14  weeks,  there  is  clear  proof  that  the  PM  should 
be  concerned  and  actions  should  be  taken  quickly  before  the  project  schedule  slips  and 
costs  increase. 
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Figure  12:  Project  C  Percent  of  Customer  Requirements  Approved  Trend  Lines 


Figure  13:  Project  C  Percent  of  Derived  Requirements  Approved  Trend  Lines 

Out  of  the  six  project  trend  lines,  four  of  them  fell  within  the  historical  bounds  at 
least  50  percent  of  the  time,  and  one  did  not  fall  within  the  bounds  at  any  time  during  the 
project.  For  Project  A’s  customer  requirements  trend  line,  having  the  trend  line  fall 
outside  of  the  bounds  should  not  be  considered  negatively.  This  simply  means  the 
project  did  not  follow  the  historical  data  but  rather  was  ahead  of  the  historical  trend. 
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Percent  Requirements  Growth  Derived  Measures 

The  trend  lines  for  percent  requirements  growth  derived  measure  are  percentages 
graphed  over  time.  This  derived  measure  takes  into  account  the  number  of  new 
requirements  versus  the  number  of  old  requirements  and  uses  the  equation: 

/number  of  new  requirements^. 

V  number  of  old  requirements  ) 

By  applying  this  derived  measures’  equation  to  the  historical  and  project  data,  a 
graph  with  multiple  spikes  is  created  (Figure  14).  This  graph  is  very  hard  for  the  PMs  to 
interpolate.  To  help  improve  the  way  the  information  is  displayed,  the  percentage 
determined  at  each  collection  interval  was  added  to  the  previous  interval  using  the 
equation: 


current  week 


I  ([( 

1=1  v 


number  of  new  requirements 
number  of  old  requirements 
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Figure  14:  Sample  Percent  of  Growth  Derived  Measures 

By  applying  the  second  equation  to  the  historical  information,  more  insight  is 
gathered  for  this  derived  measure.  The  customer  requirements  historical  bound  lines  are 
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overly  wide  for  this  derived  measure  (Figure  15).  After  week  five  the  upper  bound  is 
never  less  than  374  percent  away  from  the  historical  trend  line  with  a  maximum 
difference  of  457  percent.  The  lower  bound  is  closer  with  a  minimum  difference  of  192 
percent  and  a  maximum  difference  of  290  percent.  The  derived  requirements  historical 
bound  lines  fall  closer  to  the  historical  trend  line  but  are  still  very  wide  (Figure  16).  After 
week  five  the  upper  bound  has  a  difference  between  76  and  151  percent,  and  the  lower 
bound  has  a  difference  between  88  and  144  percent.  Having  such  wide  historical  bounds 
show  that  the  historical  projects  do  not  match  each  other  when  it  comes  to  requirements 
growth.  This  also  implies  that  this  data  is  not  reliable. 


Figure  15:  Percent  of  Customer  Requirements  Growth  Historical  Trend  Line 
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Figure  16:  Percent  of  Derived  Requirements  Growth  Historical  Trend  Line 

The  project  trend  lines  for  Project  A  do  not  match  with  the  historical  trend  lines 
and  the  derived  requirements  never  fall  within  the  historical  bounds  (Figures  17  &  18). 
The  customer  requirements  trend  line  does  fall  within  the  historical  bounds  for  83  percent 
of  the  project  but  is  still  over  100  percent  away  from  the  historical  trend  line  for  the 
majority  of  the  project.  The  project  trend  line  shows  that  the  majority  of  the  customer 
equipments  were  received  within  the  first  five  weeks  of  the  project  and  the  derived 
requirements  were  received  around  week  14.  One  reason  these  lines  do  not  match  may 
be  the  lack  of  requirements  recorded  for  the  project.  The  project  looked  at  high  level 
requirements  only  while  the  historical  data  included  all  levels  of  requirements.  Another 
reason  for  the  discrepancy  between  the  project  and  historical  lines  is  the  unreliability  of 
the  historical  information  for  this  derived  measure. 
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Figure  17:  Project  A  Percent  of  Customer  Requirements  Growth  Trend  Lines 


Figure  18:  Project  A  Percent  of  Derived  Requirements  Growth  Trend  Lines 

Project  B  trend  lines  also  do  not  follow  the  historical  trend  lines  and  neither  of  the 
trend  lines  rise  above  the  low  historical  bound  (Figures  19  &  20).  The  customer 
requirements  trend  line  and  the  lower  historical  bound  line  have  a  maximum  difference  of 
118  percent  while  the  derived  requirements  trend  line  has  a  maximum  difference  of  98 
percent  from  the  historical  trend  line.  The  customer  requirements  trend  line  only 
increases  by  seven  percent  which  leads  to  the  conclusion  that  either  there  is  the  lack  of 
requirements  and  more  are  expected  in  the  future  or  the  majority  of  the  requirements 
were  received  the  first  day  of  the  project.  Like  Project  A,  a  reason  for  the  trend  lines  not 
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matching  could  be  a  result  of  the  lack  of  requirements  recorded  for  the  project  and  the 
unreliability  of  the  historical  information  for  this  derived  measure.  Project  B  recorded 
more  requirements  than  Project  A  but  still  not  as  many  as  the  historical  projects  recorded. 
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Figure  19:  Project  B  Percent  of  Customer  Requirements  Growth  Trend  Lines 
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Figure  20:  Project  B  Percent  of  Derived  Requirements  Growth  Trend  Lines 

As  in  the  previous  two  projects,  Project  C  trend  lines  do  not  match  the  historical 
trend  lines  (Figures  21  &  22).  Also,  neither  type  of  requirement  falls  within  the  historical 
bounds.  The  customer  requirements  trend  line  stays  well  below  the  historical  bounds, 
never  increasing  more  than  27  percent.  The  derived  requirements  trend  line  spikes  to 
1800  percent  at  week  four  never  getting  closer  than  1372  percent  to  the  upper  bound  line. 
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Figure  21:  Project  C  Percent  of  Customer  Requirements  Growth  Trend  Lines 
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Figure  22:  Project  C  Percent  of  Derived  Requirements  Growth  Trend  Lines 

None  of  the  projects  trend  lines  follow  the  historical  trend  lines  or  fall  within  the 
requirements  historical  bounds,  with  the  exception  of  Project  A’s  customer  requirements, 
for  the  percent  requirements  growth  derived  measure.  The  wide  historical  bounds  also 
make  this  derived  measure  very  unreliable. 

Percent  To-Be-Determined/To-Be-Reviewed  Closure  Variance  Derived  Measures 

The  trend  lines  for  percent  TBD/TBR  closure  variance  derived  measure  are 
percentages  graphed  over  time.  This  derived  measure  takes  into  account  the  number  of 
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requirements  TBD/TBR  versus  the  number  of  TBD/TBR  requirements  that  have  been 

closed.  The  derived  measure  uses  the  equation: 

( TBD/TBR  planned  for  closure )  —  ( TBD/TBR  closed ) 

TBD/TBR  planned  for  closure 

The  historical  trend  line  suggests  that  at  no  point  during  the  project  should  the 
percent  of  TBD/TBR  customer  requirements  be  over  87  percent  and  derived  requirements 
be  over  75  percent  (Figures  23  &  24).  It  also  suggests  that  halfway  through  the  project 
the  amount  of  TBD/TBR  requirements  should  be  no  more  than  40  percent  open,  and  by 
the  time  the  project  is  three-fourths  complete  there  should  be  no  open  TBD/TBR 
requirements.  The  customer  requirements  historical  bound  is  slightly  wider  than  the 
derived  requirements  historical  bound  with  a  minimum  difference  of  seven  percent  and  a 
maximum  difference  of  36  percent  versus  a  minimum  difference  of  seven  percent  and 
maximum  difference  of  29  percent.  For  this  derived  measure,  having  the  project  trend 
lines  below  the  historical  trend  lines  is  a  positive  result.  If  the  project  trend  lines  are 
above  the  historical  trend  lines,  the  PM  should  look  into  why  TBD/TBR  requirements  are 
not  being  closed  and  if  any  actions  need  to  be  taken  to  ensure  requirements  are  being 
closed  in  an  acceptable  time  frame. 


j  x  100 
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Figure  23:  Percent  of  Customer  Requirements  TBD/TBR  Closure  Variance 

Historical  Trend  Line 


Figure  24:  Percent  of  Derived  Requirements  TBD/TBR  Closure  Variance  Historical 

Trend  Line 


Project  A’s  trend  lines  do  not  match  the  historical  trend  lines  and  rarely  stay 
within  the  historical  bounds  (Figures  25  &  26).  The  customer  requirements  trend  line 
falls  within  the  bounds  only  17  percent  of  the  time  while  the  derived  requirements  trend 
line  falls  within  the  bounds  7  percent  of  the  time.  Even  though  the  customer 
requirements  project  trend  line  does  not  follow  the  historical  information,  there  is  nothing 
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wrong  with  the  number  of  customer  requirements  the  project  has  TBD/TBR.  Having  no 
requirements  TBD/TBR  is  rare  but  a  very  good  thing.  In  contrast,  the  derived 
requirements  project  trend  line  starts  well  above  the  historical  upper  bound  with  a 
difference  of  25  percent.  The  PM  should  be  concerned  that  none  of  the  derived 
requirements  have  been  approved.  However,  around  week  13  this  concern  should  be 
lessened,  and  by  week  16  there  should  be  no  major  concern  about  the  number  of 
unapproved  derived  requirements. 
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Figure  25:  Project  A  Percent  of  Customer  Requirements  TBD/TBR  Closure 

Variance  Trend  Lines 
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Figure  26:  Project  A  Percent  of  Derived  Requirements  TBD/TBR  Closure  Variance 

Trend  Lines 


Project  B’s  trend  lines  closely  matches  the  historical  trend  lines  and  stays  within 
or  below  the  requirements  historical  bounds  (Figures  27  &  28).  The  customer 
requirements  trend  line  stays  within  the  historical  bounds  83  percent  of  the  time.  The 
derived  requirements  trend  line  stays  within  the  historical  bounds  only  32  percent  of  the 
time,  but  this  should  not  a  negatively  viewed  thing.  Having  fewer  TBD/TBR 
requirements  is  a  positive  outcome.  The  PM  should  not  be  overly  concerned  with  this 
aspect  of  the  project  and  can  concentrate  more  of  his  time  on  other  aspects  of  the  project. 
However,  if  the  project’s  derived  requirements  trend  line  continues  on  its  current  path  by 
week  15,  the  PM  should  be  concerned  and  investigate  why  the  derived  requirements  have 
not  been  determined. 
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Figure  27:  Project  B  Percent  of  Customer  Requirements  TBD/TBR  Closure 

Variance  Trend  Lines 
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Figure  28:  Project  B  Percent  of  Derived  Requirements  TBD/TBR  Closure  Variance 

Trend  Lines 


Project  C’s  trend  lines  are  close  to  the  historical  trend  lines  for  a  portion  of  the 
project,  but  also  exceed  the  upper  historical  bound  (Figures  29  &  30).  The  project 
requirements  trend  line  stays  within  the  bounds  for  73  percent  of  the  project  and  the 
derived  requirements  trend  line  for  36  percent  of  the  project.  Early  into  the  project  the 
percentage  of  the  project’s  customer  requirements  that  need  TBD  are  well  above  the 
historical  trend  line  but  still  stays  within  the  historical  bound.  The  PM  should  not  be 
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overly  concerned  with  this  portion  of  the  project  until  around  week  nine  when  the  project 
trend  line  exceeds  the  historical  bound  by  13  percent.  On  the  other  hand,  the  PM  should 
be  concerned  with  the  derived  requirements  from  the  start  of  the  project  with  the  trend 
line  exceeding  the  historical  bound  by  25  percent.  After  week  six  the  project  line  falls 
below  the  upper  historical  bound;  however,  at  week  ten  the  project  trend  line  starts  to 
exceed  the  upper  bound  by  30  percent. 
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Figure  29:  Project  C  Percent  of  Customer  Requirements  TBD/TBR  Closure 

Variance  Trend  Lines 
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Figure  30:  Project  C  Percent  of  Derived  Requirements  TBD/TBR  Closure  Variance 

Trend  Lines 
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Of  the  three  projects,  Project  B  follows  the  historical  trend  lines  the  closest  and 
stays  within  the  historical  bounds  most  often.  However,  Project  A’s  customer 
requirements  trend  line  had  the  best  outcome  with  no  TBD/TBR  requirements  during  the 
project  even  though  the  trend  line  did  not  always  fall  within  the  historical  bounds. 

Project  A  illustrates  that  not  following  the  historical  trend  line  does  not  have  to  be  a 
negative  outcome;  it  simply  indicates  that  the  project  is  not  typical  and  does  not  follows 
the  historical  information. 

Percent  Requirements  Modified  Derived  Measures 

The  trend  lines  for  percent  requirements  modified  derived  measure  are 

percentages  graphed  over  time.  This  derived  measure  takes  into  account  the  number  of 

requirements  modified  versus  the  number  of  total  requirements  and  uses  the  equation: 

/ total  number  of  requirements  modified\ 

V  total  number  of  requirements  ) 

The  historical  trend  suggests  that  a  significant  number  of  requirements  are 
modified  within  the  first  part  of  the  project  and  gradually  increases  until  the  end  (Figures 
31  &  32).  From  the  trend  data,  at  no  point  should  the  number  of  modified  requirements 
be  greater  than  35  percent.  Other  than  between  weeks  ten  and  12,  the  customer 
requirements  historical  bound  stays  within  16  percent  of  the  customer  requirements 
historical  trend  line.  During  weeks  ten  and  12,  the  upper  historical  bound  increases  to  a 
difference  of  22  percent.  This  jump  between  weeks  ten  and  12  is  due  to  a  historical  out 
layer.  The  derived  requirements  historical  bound  is  more  jagged,  but  always  stays  within 
19  percent  of  the  historical  trend  line.  For  this  derived  measure,  having  the  project  trend 
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lines  below  the  historical  trend  lines  is  a  positive  outcome.  However,  the  PM  should  be 
aware  that  a  significant  number  of  modified  requirements  could  occur  later  into  the 
project.  Having  the  project  trend  lines  above  the  historical  trend  lines  is  a  negative 
outcome.  This  suggests  that  the  project  is  unstable  and  that  the  PM  needs  to  keep  a  close 


watch  on  the  requirements. 
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Figure  31:  Percent  Customer  Requirements  Modified  Historical  Trend  Line 


Figure  32:  Percent  Derived  Requirements  Modified  Historical  Trend  Line 

The  percentage  of  modified  requirements  for  Project  A  is  highly  unusual,  and 
therefore  the  project  trend  lines  do  not  match  the  historical  trend  lines  (Figures  33  &  34). 
It  is  uncommon  to  have  so  many  of  the  requirements  modified  during  any  part  of  a 
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project.  From  the  beginning  of  the  project,  the  percent  of  requirements  that  were 
modified  jumped  well  above  the  historical  percentage  and  continued  to  climb.  By  week 
30  the  customer  requirements  trend  line  is  164  percent  above  the  customer  requirements 
historical  upper  bound  line,  and  the  derived  requirements  trend  line  is  429  percent  above 
the  derived  requirements  historical  upper  bound  line.  The  given  data  suggests  this  is  an 
extremely  unstable  project  and  the  PM  should  be  highly  concerned. 
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Figure  33:  Project  A  Percent  Customer  Requirements  Modified  Trend  Lines 
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Figure  34:  Project  A  Percent  Derived  Requirements  Modified  Trend  Lines 


Project  B’s  project  trend  lines  also  do  not  match  the  historical  trend  lines,  but  is 
well  within  the  requirements  historical  bounds  for  100  percent  of  the  project  (Figures  35 
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&  36).  The  PM  of  this  project  should  not  be  concerned  about  the  number  of  requirements 
that  have  been  modified.  The  first  half  of  the  project  was  very  stable  and  had  no 
modified  requirements.  During  week  12,  requirements  started  to  be  modified  and 
exceeded  the  historical  percentages  but  stayed  within  the  requirements  historical  bounds. 
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Figure  35:  Project  B  Percent  Customer  Requirements  Modified  Trend  Lines 
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Figure  36:  Project  B  Percent  Derived  Requirements  Modified  Trend  Lines 

Project  C  is  similar  to  Project  B  where  no  requirements  are  modified  until  late 
into  the  project,  and  both  trend  lines  stay  within  the  historical  bounds  for  100  percent  of 
the  project  (Figures  37  &  38).  This  project  does  not  receive  any  modified  customer 
requirements  until  the  project  is  over  70  percent  complete,  and  there  currently  are  no 
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modified  derived  requirements.  This  is  a  very  stable  project.  The  information  gives  the 
PM  insight  into  what  the  general  trend  of  a  project  could  be  and  increases  the 


expectations  that  more  requirements  might  be  modified  before  the  end  of  the  project. 
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Figure  37:  Project  C  Percent  Customer  Requirements  Modified  Trend  Lines 
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Figure  38:  Project  C  Percent  Derived  Requirements  Modified  Trend  Lines 


Both  Project  B  and  C  stayed  within  the  requirements  historical  bounds  and  did  not 
receive  any  modified  requirements  until  late  into  their  projects.  Project  A  was  not  close 
to  the  requirements  historical  bounds  due  to  the  rarity  of  how  many  modified 
requirements  were  involved. 
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Estimated  Impact  of  Requirements  Change  Derived  Measures 

The  trend  lines  for  the  estimated  impact  of  requirements  change  derived  measure 
are  summations  over  time.  This  derived  measure  takes  into  account  the  number  of 
overtime  hours  worked  during  the  entire  project.  The  type  of  requirements  for  this 
derived  measure  was  not  split  into  customer  and  project  requirements  due  to  the  type  of 
information  to  which  the  PMs  had  access.  The  PMs  only  have  easy  access  to  the  amount 
of  overtime  worked  for  their  project  each  week.  They  do  not  have  easy  access  as  to 
which  requirement  caused  the  overtime.  Obtaining  this  information  would  take  more 
time  than  they  have  available.  Overtime  hours  have  multiple  causes  and  are  not  limited 
to  requirements  change.  This  derived  measure  should  be  combined  with  other  derived 
measures,  such  as  the  percent  of  modified  requirements  derived  measure,  for  the  data  to 
be  valid. 

The  historical  trend  suggests  that  the  majority  of  overtime  is  worked  toward  the 
end  of  a  project  and  spikes  significantly  during  the  last  few  weeks  (Figure  39).  The 
historical  bounds  range  between  six  and  18  hours  above  the  historical  trend  line  and 
between  two  and  14  hours  below.  A  large  number  of  overtime  hours  occurring  at  the 
beginning  of  a  project  is  not  common  and  needs  to  be  investigated  by  the  PM. 
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Figure  39:  Estimated  Impact  of  Requirements  Change  Historical  Trend  Line 

The  amount  of  overtime  used  for  Project  A  starts  off  high  and  increases  as  the 
project  progresses  (Figure  40).  This  links  directly  back  to  the  number  of  modified 
requirements  recorded  with  the  previous  derived  measure,  percent  requirements 
modified.  By  week  30,  the  number  of  hours  worked  on  Project  A  exceeds  the  historical 
bounds  by  242  hours.  The  PM  should  be  very  concerned  about  the  amount  of  overtime 
being  used  and  should  take  corrective  actions  if  possible  along  with  notifying  the 
customer  that  there  will  most  likely  be  a  cost  overrun  and/or  a  project  schedule  extension. 
The  derived  measure  shows  that  this  much  overtime  is  not  normal  and  that  the  amount  of 
overtime  is  likely  to  continue  to  increase  during  the  remainder  of  the  project. 
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Figure  40:  Project  A  Estimated  Impact  Requirements  Change  Trend  Lines 

The  project  trend  line  for  Project  B  is  close  to  the  historical  trend  line  and  stays 
within  historical  hour  bounds  for  the  entire  project  (Figure  41).  The  project  did  not 
require  overtime  hours  until  week  12,  which  is  consistent  with  the  point  at  which  the 
requirements  started  to  be  modified.  Although  the  amount  of  overtime  for  the  project  at 
week  12  is  more  than  double  the  amount  of  the  historical  data,  the  PM  should  not  be 
overly  concerned  because  it  is  still  within  the  historical  bounds.  However,  the  PM  should 
keep  an  eye  on  the  amount  of  overtime  accumulated  and  expect  to  incur  additional 


overtime  towards  the  end  of  the  project. 


Figure  41:  Project  B  Estimated  Impact  Requirements  Change  Trend  Lines 
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Project  C  did  not  incur  any  overtime;  therefore,  its  project  trend  line  does  not 
follow  the  historical  trend  line  but  stays  within  the  historical  bounds  (Figure  42).  The 
derived  measure  informs  the  PM  that  there  is  a  high  probability  that  overtime  will  be 
used  before  the  end  of  the  project  and  that  the  PM  should  expect  the  majority  of  the 
overtime  to  occur  after  the  12th  week. 
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Figure  42:  Project  C  Estimated  Impact  Requirements  Change  Trend  Lines 

For  the  information  provided  in  this  derived  measure  to  be  valid,  it  must  be 
combined  with  other  derived  measures.  This  is  due  to  the  fact  that  overtime  is  the  result 
of  multiple  factors  and  is  not  limited  to  requirements  change.  For  these  three  projects, 
the  derived  measure  links  back  to  the  percent  requirements  modified  derive  measure.  As 
requirements  are  modified,  the  PM  should  expect  some  amount  of  overtime  to  be  used. 
Except  for  Project  A,  the  project  trend  lines  fell  within  the  historical  hour  bounds  for  100 
percent  of  the  project  and  the  projects  results  were  favorable.  This  derived  measure  gives 
valuable  insight  at  the  beginning  of  the  project  to  budget  for  extra  overtime  in  case  it  is 
needed. 
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Requirement  Validation  Rate  Derived  Measure 

The  trend  lines  for  the  requirement  validation  rate  derived  measure  takes  into 
account  the  length  of  the  time  it  takes  for  a  requirement  to  be  validated  in  terms  of  weeks. 
For  example,  the  amount  of  requirements  that  were  validated  within  one  week  of 
receiving  the  requirement  was  calculated.  This  derived  measure  is  determined  by  using 
the  equation: 

(date  requirement  added )  —  ( date  requirement  validated ) 

7  days 

Utilizing  the  equation  above,  the  result  depends  heavily  on  the  amount  of  requirements 
recorded  in  the  historical  and  current  projects  (Figure  43).  The  amount  of  requirements 
varies  greatly  between  the  different  projects  making  it  hard  to  compare  the  different  trend 
lines.  The  graphs  for  this  derived  measure  also  only  contain  information  for  requirements 
that  have  already  been  validated.  As  more  requirements  are  validated,  the  entire  shape  of 
the  graph  may  change.  To  make  it  easier  to  compare  the  different  trends  lines  the  percent 
of  requirements  validated  over  a  period  of  time  was  calculated.  For  example,  the  percent 
of  requirements  that  were  validated  within  one  week  of  receiving  the  requirement  was 
calculated.  The  new  equation  is: 
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Figure  43:  Requirements  Validation  Rate  Derived  Measure 

The  data,  using  the  second  equation,  suggests  that  the  majority  of  customer 
requirements  should  be  validated  within  five  weeks  of  obtaining  the  requirement,  and  no 
customer  requirement  should  take  more  than  13  weeks  (Figure  44).  Derived  requirement 
should  not  take  more  than  20  weeks  to  be  validated  (Figure  45).  The  historical  data  also 
suggest  that  derived  requirements  take  longer  to  validate  than  customer  requirements. 
The  historical  data  had  an  average  total  number  of  45  customer  requirements  and  62 


derived  requirements.  The  ideal  project  trend  line  would  be  above  the  historical  trend. 


Figure  44:  Customer  Requirements  Validation  Rate  Historical  Trend  Lines 
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Figure  45:  Derived  Requirements  Validation  Rate  Historical  Trend  Lines 

By  week  30,  60  percent  into  the  project  timeline,  all  1 1  of  Project  A’s 
requirements  had  been  validated.  Of  these  1 1  requirements,  six  are  customer 
requirements  and  five  are  derived  requirements.  The  customer  requirements  took  five, 
six  and  10  weeks,  respectively,  to  be  validated  and  the  trend  line  stayed  within  the 
historical  bounds  60  percent  of  the  time  for  the  25  weeks  (Figure  46).  The  project’s 
derived  requirements  took  six,  eight  and  15  weeks,  respectively,  to  be  validated  and  the 
trend  line  stayed  within  the  historical  bounds  40  percent  of  the  time  for  the  25  weeks 
(Figure  47).  For  both  types  of  requirements  the  project  trend  lines  are  below  the 
historical  trend  lines  and  the  historical  bounds  until  all  the  project  lines  reach  100 
percent.  Even  though  the  data  shows  the  trend  lines  within  the  bounds  60  percent  and  40 
percent  of  the  25  weeks,  the  amount  of  time  it  is  taking  to  validate  requirements  is  longer 
than  historically  shown.  The  data  indicates  an  unstable  project  and  the  PM  should 
investigate  as  to  why  it  is  taking  longer  than  normal  for  the  requirements  to  be  validated. 
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Figure  46:  Project  A  Customer  Requirements  Validation  Rate  Trend  Lines 


Figure  47:  Project  A  Derived  Requirements  Validation  Rate  Trend  Lines 

Project  B  had  more  requirements  recorded  than  Project  A.  Out  of  the  26  recorded 
requirements,  16  were  customer  requirements  and  ten  were  derived  requirements.  By 
week  12,  60  percent  into  the  project  timeline,  15  of  the  customer  requirements  and  eight 
of  the  derived  requirements  were  approved.  The  customer  requirements  took  two,  four, 
six,  eight,  nine  and  ten  weeks,  respectively  (Figure  48).  The  derived  requirements  took 
four  and  nine  weeks,  respectively,  to  be  validated  (Figure  49).  The  customer 
requirements  fell  within  the  historical  bounds  60  percent  of  the  time  and  derived 
requirements  trend  lines  fell  within  the  historical  bounds  64  percent  of  the  time  for  the  25 
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weeks.  Similarly  to  Project  A,  both  types  of  requirements  trend  lines  are  below  the 
historical  trend  lines,  and  even  though  the  data  show  the  trend  lines  within  the  bounds 
over  60  percent  of  the  25  weeks,  the  amount  of  time  it  is  taking  to  validate  requirements 
is  longer  than  historically  shown.  The  data  indicates  an  unstable  project,  and  the  PM 
should  investigate  why  it  is  taking  longer  than  normal  for  the  requirements  to  be 
validated. 


Figure  48:  Project  B  Customer  Requirements  Validation  Rate  Trend  Lines 


Figure  49:  Project  B  Derived  Requirements  Validation  Rate  Trend  Lines 

With  a  total  number  of  42  requirements,  Project  C  comes  closest  to  the  amount  of 
requirements  from  the  historical  information  out  of  the  three  projects;  however,  the 
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length  of  the  project  is  the  shortest.  Twenty-two  of  the  requirements  are  customer 
requirements  and  20  are  derived  requirements.  By  week  1 1,  78  percent  into  the  project 
timeline,  18  of  the  customer  requirements  and  1 1  of  the  derived  requirements  were 
validated.  The  customer  requirements  took  one  through  three  weeks  to  be  validate  and 
the  derived  requirements  took  one  through  four  weeks  to  be  validate  (Figures  50  &  51). 
The  customer  requirements  stay  within  the  bounds  96  percent  of  the  25  weeks  and  the 
derived  requirements  stayed  within  bounds  72  percent  of  the  25  weeks.  The  customer 
requirements  start  inside  the  requirements  historical  bounds  and  continue  to  increase  until 
they  exceeded  the  bounds.  The  derived  requirements  start  above  the  requirements 
historical  bounds  and  remain  near  the  top  of  the  historical  bound  or  above  only  going 
inside  the  bounds  by  15  percent.  Unlike  the  previous  two  projects,  this  data  shows 
Project  B  is  stable. 


Figure  50:  Project  C  Customer  Requirements  Validation  Rate  Trend  Lines 
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Figure  51:  Project  C  Derived  Requirements  Validation  Rate  Trend  Lines 

Out  of  all  three  projects,  Project  C  was  the  only  one  that  came  close  to  fitting  the 
historical  lines.  One  reason  for  this  is  that  Project  C  has  the  most  recorded  requirements 
and  the  closest  to  the  amount  of  recorded  requirements  in  the  historical  data.  Another 
reason  for  only  one  out  of  the  three  projects  matching  the  historical  trends  is  that  this 
derived  measure  only  took  into  account  the  length  of  time  it  took  for  a  requirement  to  be 
validated  and  not  the  length  of  the  project  or  the  number  of  requirements.  More  research 
needs  to  be  conducted  for  this  derived  measure.  The  effect  of  the  different  project 
lengths  needs  to  be  researched  along  with  the  difference  in  the  number  of  requirements. 
Project  length  and  amount  of  requirements  should  be  scaled. 


Percent  Requirements  Validated  Derived  Measures 

The  trend  lines  for  the  percent  requirements  validated  derived  measure  are 

percentages  graphed  over  time.  This  derived  measure  takes  into  account  the  number  of 

validated  requirements  versus  the  total  number  of  requirements  and  uses  the  equation: 

/  requirements  validated  \ 

- 7~ - : - 7 -  x  100 

\total  number  of  requirements / 
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The  historical  trend  lines  suggest  that  no  less  than  13  percent  of  customer 
requirements  and  25  percent  of  derived  requirements  should  be  validated  at  any  point 
during  the  project  (Figures  52  &  53).  By  the  time  the  project  is  50  percent  complete, 
both  types  of  requirements  should  be  no  less  than  60  percent  validated,  and  by  the  time 
the  project  is  75  percent  complete  all  of  the  requirements  should  be  validated.  The 
historical  bound  for  the  customer  requirements  is  very  wide  for  most  of  the  project  with  a 
maximum  lower  difference  of  39  percent  and  maximum  upper  difference  of  42  percent. 

In  contrast,  the  historical  bound  for  the  derived  requirements  is  narrower  with  maximum 
lower  difference  of  16  percent  and  maximum  upper  difference  of  29  percent.  For  this 
derived  measure,  having  the  project  trend  lines  above  the  historical  trend  lines  is  a 
positive  outcome.  Having  the  project  trend  lines  below  the  historical  trend  lines  is  a 
negative  outcome.  The  PM  needs  to  examine  why  requirements  are  not  being  validated 
and  to  determine  if  action  needs  to  be  taken  to  ensure  the  requirements  are  being 


validated  in  a  timely  manner. 


Figure  52:  Percent  Customer  Requirements  Validated  Historical  Trend  Line 
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Figure  53:  Percent  Derived  Requirements  Validated  Historical  Trend  Line 

Project  A’s  customer  requirements  trend  line  start  within  the  historical  bounds 
and  continue  to  grow  until  it  exceeds  the  bounds  (Figure  54).  It  only  stays  within  the 
bounds  for  56  percent  of  the  project.  However,  this  is  not  a  negative  outcome  for  the 
project.  The  project  trend  does  not  follow  the  historical  norm  but  instead  shows  a  better 
result.  The  derived  requirements  trend  does  not  start  within  the  historical  bounds  and  is 
only  within  the  bounds  17  percent  of  the  project  (Figure  55).  For  the  first  eight  weeks, 
the  project  line  is  no  less  than  25  percent  under  the  lower  historical  bound  and  barely 
rises  above  the  lower  bound  by  four  percent  in  week  10.  The  PM  should  be  very 
concerned  about  the  time  it  is  taking  to  validate  the  derived  requirements  until  week  16 
when  the  project  line  shift  sharply  upwards  above  the  upper  historical  bound. 
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Figure  54:  Project  A  Percent  Customer  Requirements  Validated  Trend  Lines 
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Figure  55:  Project  A  Percent  Derived  Requirements  Validated  Trend  Lines 

Project  B’s  trend  lines  match  the  percent  requirements  validated  derived  measure 
trend  lines  fairly  well  (Figures  56  &  57).  For  the  majority  of  the  project,  the  project  trend 
lines  are  slightly  above  the  historical  trend  lines  and  stay  within  the  historical  bounds. 

The  customer  requirements  trend  line  stays  within  the  bound  for  92  percent  of  the  project, 
and  the  derived  requirements  trend  line  stays  within  the  bounds  for  58  percent  of  the 
project.  At  week  ten,  when  the  project  trend  lines  fall  below  the  historical  trend  lines,  the 
PM  should  not  be  overly  concerned  about  the  direction  of  the  trend.  The  derived 
requirements  project  trend  line  is  still  within  the  historical  bounds,  and  the  customer 
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requirements  project  trend  line  is  only  6  percent  below  the  historical  bound  and  has  an 
upward  trend.  This  is  proven  by  week  12,  when  the  trend  lines  match  with  the  historic 


trend  lines. 


Figure  56:  Project  B  Percent  Customer  Requirements  Validated  Trend  Lines 


Figure  57:  Project  B  Percent  Derived  Requirements  Validated  Trend  Lines 

Project  C’s  customer  requirements  trend  line  stays  within  the  historical  bounds  for 
64  percent  of  the  project  (Figure  58).  After  week  nine,  when  not  all  of  the  customer 
requirements  have  been  validated,  the  PM  should  be  concerned  and  investigate.  Project 
C’s  derived  requirements  trend  line  does  not  stay  within  the  bounds  and  instead  remains 
below  the  lower  bound  (Figure  59).  The  spike  on  the  graph  at  week  three  should  be 
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considered  an  anomaly  and  be  ignored.  This  spike  links  back  to  the  percent  requirements 
growth  derived  measure  when  the  project’s  requirements  jumped  by  23  percent  in  week 


four  creating  new  requirements  which  needed  to  be  validated. 
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Figure  58:  Project  C  Percent  Customer  Requirements  Validated  Trend  Lines 
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Figure  59:  Project  C  Percent  Derived  Requirements  Validated  Trend  Lines 

Project  B’s  trend  lines  were  the  closest  out  of  the  three  projects  to  match  the 
historical  trend  lines  and  stay  within  the  historical  bounds.  On  the  other  hand,  Project  A 
and  C’s  derived  requirement  trend  lines  were  not  close  to  the  historical  trend  line  and,  for 
the  majority  of  the  projects,  did  not  stay  within  the  historical  bounds. 
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Summary 

By  conducting  research  and  applying  results  from  a  questionnaire,  the  top  three 
SE  processes  for  a  high  speed  sled  track  environment  were  determined.  These  three  SE 
processes  include  requirements  management,  risk  management  and  technical  planning. 
Using  these  findings  and  a  second  questionnaire,  the  top  three  LI  trends  for  a  high  speed 
sled  track  environment  were  determined.  These  three  LI  trends  include  requirements, 
requirements  validation,  and  facility  and  equipment  availability.  From  the  top  three  LI 
trends,  requirements  and  requirements  validation  trends  were  chosen  to  be  broken  down 
and  the  current  and  historical  project  trends  compared. 

From  the  two  LI  trends,  seven  derived  measures  were  chosen  to  obtain  historical 
and  project  data  to  be  analyzed:  percent  requirements  approved,  percent  requirements 
growth,  percent  requirements  TBD/TBR  closure  variance  per  plan,  percent  requirement 
modified,  estimated  impact  of  requirements  change,  requirements  validation  rate,  and 
percent  requirements  validated.  Each  of  the  derived  measures  is  unique.  Obtaining 
information  from  these  derived  measures  gives  an  understanding  of  how  a  project  is 
going  to  be  effective  in  the  future.  The  historical  bounds  used  for  this  thesis  are  the 
maximum  and  the  minimum  historical  values  for  each  collection  interval.  If  these 
historical  bounds  are  overly  wide,  as  shown  in  the  percent  requirement  growth  derived 
measure,  the  historical  projects  do  not  agree  for  that  derived  measure  and  information 
obtain  is  not  reliable. 

For  the  majority  of  the  derived  measures,  the  project  trend  lines  did  not  match  the 
historical  trend  lines.  Thirty  percent  of  the  project  trend  lines  were  located  within  the 
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historical  trend  for  over  70  percent  of  the  project,  30  percent  of  the  project  trend  lines 
were  located  within  the  historical  bounds  between  40  percent  and  60  percent  of  the 
project  and  40  percent  of  the  project  trend  lines  were  located  within  the  historical  bounds 
for  less  than  39  percent  of  the  project  (Figure  60).  There  could  be  many  reasons  for  the 
discrepancy  between  the  current  projects  and  historical  trend  lines.  Among  these  reasons 
are  the  number  of  requirements  being  tracked  by  the  PM,  the  communication  between  the 


PM  and  the  customer,  the  length  of  the  projects,  and  the  fact  that  no  project  is  truly  the 
same  as  another. 


#/o  Within 
Historical  Bounds 
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Figure  60:  Percent  of  Time  Project  Lines  were  Inside  of  Historical  Bounds 

A  project  trend  line  not  lying  within  the  historical  bounds  does  not  automatically 
constitute  a  negative  outcome.  This  only  means  the  project  is  not  following  what  has 
historically  happened.  In  some  cases  not  following  the  historical  trend  or  staying  within 
the  historical  bounds  is  a  positive  outcome.  A  good  example  is  Project  A’s  customer 
requirements  trend  for  the  percent  requirements  TBD/TBR  closure  variance  derived 
measure.  The  project’s  trend  line  is  within  the  historical  bounds  for  only  17  percent  of 
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the  project,  but  the  project  receives  no  requirements  that  need  TBD  or  TBR  which  is  a 
very  positive  outcome  for  the  project. 

For  the  information  obtained  from  the  different  LI  trends  to  be  interpolated 
properly,  the  user,  normally  a  PM  or  Systems  Engineer,  must  be  familiar  with  the  project 
or  system.  These  trend  lines  are  only  showing  part  of  the  story 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  will  cover  the  results  of  determining  which  SE  activities  the  sled 
track  community  currently  emphasizes  and  uses,  which  LI  trends  are  most  relevant  to  a 
sled  track  environment;  how  current  trend  lines  compare  to  historical  trend  lines;  and 
which  of  the  suggested  derived  measures  are  relevant  to  a  sled  track  environment. 
Recommendations  on  how  to  proceed  in  future  research  will  also  be  discussed  in  this 
chapter. 

Conclusions  of  Research 

The  DAG  describes  16  key  SE  processes  that  should  be  utilized  during  a  project. 
By  researching  HHSTT  project  notes,  SOIs  and  various  project  meeting  minutes,  it  was 
determined  that  of  the  16  processes,  only  six  were  being  utilized  at  the  HHSTT: 
requirements  management,  risk  management,  interface  management,  stakeholder 
requirements  definition,  verification  and  validation  processes.  A  questionnaire  was  given 
to  members  of  the  HHSTT  regarding  which  processes  were  most  useful  to  them.  In  the 
first  part  of  the  questionnaire,  members  of  the  HHSTT  rated  the  top  processes.  The  top 
processes  were  selected  by  their  average  rating.  These  top  processes  included:  technical 
planning,  requirements  management,  risk  management,  decision  analysis  and  technical 
assessment.  In  the  second  part  of  the  questionnaire,  members  of  the  HHSTT  ranked  the 
entire  group  of  processes  as  to  usefulness.  The  top  processes  were  selected  by  their 
average  ranking  and  included:  requirements  management,  technical  planning,  risk 
management  and  data  management.  Combining  information  gathered  from  research  and 
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the  questionnaire,  the  top  three  most  useful  SE  processes  in  the  sled  track  community  are 
requirements  management,  risk  management  and  technical  planning  processes. 

All  of  the  16  SE  processes  link  to  at  least  one  of  the  LI  trends.  The  six  currently 
utilized  SE  processes  link  to  15  LI  trends.  These  trends  include  requirements,  systems 
definition  change  backlog,  requirements  validation,  requirements  verification,  work 
product  approval,  review  action  closure,  risk  exposure,  risk  treatment,  technical 
measurement,  systems  engineering  staffing  and  skill,  process  compliance,  facility  and 
equipment  availability,  defect/error,  system  affordability,  and  schedule  and  cost  pressure. 
Taking  into  account  information  gathered  from  the  SE  process  questionnaire,  the  number 
of  linked  LI  trends  drops  to  ten.  These  trends  include  requirements,  work  product 
approval,  review  action  closure,  risk  exposure,  risk  treatment,  systems  engineering 
staffing  and  skill,  processes  compliance,  facility  and  equipment  availability,  defect/error, 
and  scheduled  cost  pressure.  A  second  questionnaire  was  given  to  members  of  the 
HHSTT  regarding  which  LI  trends  were  thought  to  be  most  useful.  The  first  part  of  the 
questionnaire  rated  each  of  the  1 8  LI  trends  as  to  their  individual  usefulness.  The  top  LI 
trends  were  selected  by  their  average  rating.  The  top  LI  trends  were  identified  as: 
requirements,  requirements  validation,  facility  and  equipment  availability,  and 
requirements  verification.  The  second  part  of  the  questionnaire  ranked  each  of  the  1 8  LI 
trends  as  a  group  to  their  usefulness.  The  top  LI  trends  were  selected  by  their  average 
ranking  and  included:  requirements,  requirements  validation,  facility  and  equipment 
availability,  and  requirements  verification.  Combining  the  information  obtained  from  the 
SE  processes  research  and  the  LI  trends  questionnaire,  the  top  three  most  relevant  LI 
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trends  for  a  sled  track  environment  are  requirements,  facility  and  equipment  availability, 
and  requirements  validation. 

The  requirements  trend  and  the  requirements  validation  trend  were  chosen  to 
conduct  project  and  historical  trend  research.  Both  of  the  LI  trends  were  broken  down 
into  their  suggested  derived  measures.  Five  out  of  the  nine  suggested  derived  measures 
for  the  requirements  trend  were  used,  and  both  of  the  suggested  derived  measures  for  the 
requirements  validation  trend  were  used.  Overall,  the  current  project  trends  did  not 
match  the  historical  trends.  Thirty  percent  of  the  project  trend  lines  were  located  within 
the  historical  trend  for  over  70  percent  of  the  project;  30  percent  of  the  project  trend  lines 
were  located  within  the  historical  bounds  between  40  percent  and  60  percent  of  the 
project;  and  40  percent  of  the  project  trend  lines  were  located  within  the  historical  bounds 
less  than  39  percent  of  the  project.  There  could  be  many  reasons  behind  the  discrepancy 
between  current  project  and  historical  trend  lines.  Among  these  reasons  are  the  number 
of  requirements  being  tracked  by  the  PM,  the  communication  between  the  PM  and  the 
customer,  the  length  of  the  projects  and  the  fact  that  no  project  is  truly  the  same  as 
another. 

A  project  trend  line  not  lying  within  the  historical  bounds  does  not  automatically 
constitute  a  negative  outcome.  This  only  means  the  project  is  not  following  what  has 
historically  happened.  In  some  cases,  not  following  the  historical  trend  or  staying  within 
the  historical  bounds  is  a  positive  outcome.  For  the  information  obtained  from  the 
different  LI  trends  to  be  interpolated  properly,  the  user,  normally  a  PM  or  Systems 
Engineer,  must  be  familiar  with  the  project  or  system.  These  trend  lines  are  only 
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showing  part  of  the  story.  The  correct  type  and  number  of  LI  trends  and  derived 
measures  need  to  be  chosen.  The  correct  collection  interval  also  needs  to  be  determined. 
There  must  be  a  balance  between  the  amount  and  type  of  information  collected,  the 
amount  of  effort  put  into  the  data  collection,  and  the  type  and  size  of  project  or  system. 

Significance  of  Research 

SE  is  utilized  at  very  low  levels  in  the  sled  track  community  and  in  most  cases  not 
emphasized  at  all.  The  use  of  LI  introduces  a  tool  to  help  promote  the  use  of  SE  and  help 
change  the  mindset  of  the  usefulness  of  SE  in  the  testing  community.  One  of  the  reasons 
SE  is  being  used  at  relatively  low  levels  is  the  lack  of  confidence  in  the  SE  process  and 
resistance  to  change.  This  thesis  has  shown  that  SE  is  currently  being  utilized  in  the  sled 
track  environment  even  though  the  sled  track  does  not  promote  its  use  or  even  recognize 
the  degree  to  which  they  are  currently  using  SE.  This  thesis  also  demonstrates  that  there 
are  LI  trends  relevant  to  the  high  speed  sled  track  community  and  that  by  utilizing  these 
trends  during  a  project  the  sled  track  can  contribute  valuable  knowledge  about  a  sled  test 
project  to  improve  the  outcome  of  the  project. 

Recommendations  for  Future  Research 

More  research  should  be  conducted  on  LI  trends  not  only  in  the  sled  test 
community  but  also  in  the  testing  community  as  a  whole.  Data  was  only  collected  from 
the  HHSTT,  and  only  two  LI  trends  were  broken  down  into  their  derived  measures.  Data 
should  be  collected  from  other  sled  tracks  as  well  as  from  other  organizations  within  the 
testing  community.  Additional  LI  trends  should  also  be  researched  and  broken  down  at 
multiple  testing  organizations.  Comparing  the  results  from  different  testing  organizations 
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will  give  a  more  complete  picture  of  which  SE  actives  the  testing  community  emphasizes, 
values  and  uses  along  with  what  LI  trends  the  testing  community  finds  most  relevant. 

Three  very  different  projects  were  used  in  this  study  to  obtain  a  wide  range  of 
data.  Project  A  is  a  50  week  project  which  consists  of  1 1  top  level  requirements,  Project 
B  is  a  40  week  project  which  consists  of  25  top  and  medium  level  requirements,  and 
Project  C  is  a  14  week  project  which  consists  of  42  requirements  from  all  levels.  Future 
research  should  be  conducted  with  more  similar  projects  to  determine  if  there  is  a  link 
between  the  usefulness  of  a  LI  trend  and  the  number  of  requirements  or  the  length  of  the 
project. 

Summary 

This  research  effort  focused  on  applying  the  SE  processes  through  the  use  of  LI 
trends  to  a  sled  track  testing  environment  to  determine  if  the  use  of  LI  trends  can  help 
improve  project  progress  and  outcomes.  The  first  step  determined  the  top  SE  processes 
for  a  high  speed  test  track  environment.  They  were  then  linked  to  the  LI  trends  to  help 
determine  the  top  LI  trends.  Two  LI  trends  were  then  chosen  to  be  applied  to  three  sled 
track  test  projects  at  the  HHSTT.  The  last  step  was  a  comparison  of  the  historical  trend 
lines  to  project  trend  lines. 

It  is  hoped  that  through  this  research  of  application  of  the  LI  trends,  the  HHSTT 
will  continue  to  utilize  LI  trends  and  place  more  emphasis  on  the  SE  processes  during 
their  projects. 
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Appendix  A 


Leading  Indicator  Measurement  Specifications 


UJ  i  tsfiiiiiniiiTiioftii  Al 

Information  Need  Description 

I  rnfio  rma  li  on 

Need 

Specifies  wh*t  the  infbr motion  reed  is  th^t  drives  why  n/e  need  this  tending 
indicator  to  mate  decisions 

Information 

Category 

Specifies  what  categofies  fas  defined  in  the  PSM)  are  app:icabie  for  this 
feading  indicator  (fbr  example,  schedule  and ptogiessy  resources  and  cost- 
product  lice  etna  stabrfrty^  product  quafity-  process  performance*,  technology 
effectiveness,.  and  customer  satisfaction) 

Measurable 

Concept 

Measurable  Concept  and  Leading  Insight 

Defines  specificity  what  is  measurable 

Leading  Insight 
Provided 

Specifies  what  specific  insights  that  the  ieodmg  indicator  may  provide  in 
context  of  the  mmsurabie  concept  -  typicatfy  a  fist  of  several  or  nxue 

Base  Measure  Specification 

Base  Measures 

A  list  cf  tbs  base  rt-^suies  that  me  used  fie  compute  one  or  mote  leading 
indicators  -  a  base  measute  is  a  single  attribute  defined  by  a  specified 
measurement  method 

Measurement 

Methods 

For  each  base  measure;  desenbes  the  method  used  to  count  the  base 
measure,  for  example  simple  counting  or  counting  then  normalized 

Unit  of 

Measurement 

Describes  the  unit  of  measure  for  each  of  the  base  measures 

Entities  and  Attributes 

Relevant  Entities 

Describes  one  or  more  particular  entities  lelevant  fbrm  this  indicator-  -  the 

object  is  to  be  measured  ffbr  example*  requirement  or  inte.tface) 

Lists  the  subset  of particular  attributes  {'characteristics  or- prope'tres)  for 
each  entity  that  ate  of  interest  form  this  leading  indicator ■ 

Derived  Measure 

Derived  Measure  Specification 

Describes  ons  or  mere  measures  that  may  be  derived  from  base  measures 
that  will  be  used  individually  or  in  combination  as  leading  indicatots 

Measurement 

Function 

The  function  for  computing  the  derived  measure  horn  the  base  measu/es 

Indicator  Specification 

A  detailed  specific  description  and  dispfay  of  the  tending  indicator:-  including 
what  base  and/of  derived  measures  are  used 

Thresholds  and 
Outliers 

Would  describe  thresholds  and  outiiers  for  the  indicator;  this  information 
would  be  company  fond possibly  pr  oject)  specific 

Decision  Criteria 

Provides  base  guidance  fbrm  trigger s  for  investigation  and  when  possrbie 
action  to  be  taken 

Indicator 

Interpretation 

p - 

Provides  some  insight  into  how  the  indicator  shouid  be  interpreted;  each 
or  ganization  would  be  expe  cted  to  tailor  this 
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Additional  Information 


Name  of  Leadinq  Indicator' 


Assumptions 


Additional 

Analysis 

Guidance 


Implementati-on 

Considerations 


User  of 
Information 


Data  Collection 
Procedure 


Data  Analysis 
Procedure 


Usts  re^tod  processes  end  sub-processes 


L'Sti-  essunptions  for  the  /ending  indtcetar  to  be  used:  for  thef  t  <=t 

fequtiements  d*tab*se  sn Teinteined 


e  on  inrpAenienting  or  using  the  indicafo/s 


Considerations  on  how  tu  injpfement  the-  indicator  (assume  this  ewp^nds  mtfi 
use  by  oroansza  ton] 


Lists  the  roiefs)  that  use  the  A ?ad 


Deters  the  placed  we  for  date  coAiection 


Deters  the  placed  we  fbransfyzing  the  date?  ptidrto  /ntetpteih *bon 


Data  Collection  Procedure  (for  each  Base  Measure) 

Complete  this  section  for  each  hose  measure  listed  in  each 
measurement  information  specification 


Frequency  of  Collect  at  least  monthly:  moss  frequently  during  peak  activity  periods.  Do 

Data  Collection  rot  sample  -  collect  dl  equipments  data. 


Responsible  Measu  lenient  Analyst,  Requirements  Manager..  Configuration  Management 

I  nd  iv  id  ua  I  M  anagei 


Activity  in  which  From  concept  and  syster"  definition  though  system  deployment 

Collected 


Potential  Sources  Requirements  Database,  Change  Boa  id  ecoios,  defect  data 
of  Data 


Verification  and  Check  data  against  Configuration  Management  records. 

Validation 


Reposi  tory  for  User  defined. 
Collected  Data 


Data  Analysis  Procedure  (for  each  Indicator' 


Biweekly  to  monthly,  depending  on  the  level  of  activity 


Typical  Tools  Requirement  Database.  Cbnnguiation  Management  Database 

Used  in  Data 

Collection  _ 


Responsible 

Individual 


MeaEL lenient  Analyst 


Review,.  Report,, 
or  User 


From  concept  and  syster-  definition  thiough  s-ysiem  deploy ment 


Requirements  Database,  Change  Boa  id  lecoicb,  defect  data 


Spieadsneet,  statistics  analysi 


Chief  SE.  Product  Manager. 
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Appendix  B 


System  Engineering  Processes  Questionnaire 


On  the  attached  page  there  is  a  list  of  16  system  engineering  processes  and  a  brief  description  according 
to  the  Defense  Acquisition  Handbook.  These  processes  are  used  to  help  keep  a  program  on  schedule  and  under 
budget.  Sled  track  projects  are  very  unique  and  do  not  always  follow  the  typical  DoD  format.  Please  look  at 
each  process  individually  and  rank  them  1  to  5  on  how  useful  they  would  be  for  a  sled  test  project,  with  1  being 
the  most  useful.  Now  look  at  them  as  a  group  and  rank  them  1  to  16  with  1  being  the  most  useful  process  for  a 
sled  test  project. 

Processes  Rank  Individually  1  -5  Rank  as  a  Group  1-16 

Decision  Analysis  Process  _  _ 

Technical  Planning  Process  _  _ 

Technical  Assessment  Process  _  _ 

Requirements  Management  Process  _  _ 

Risk  Management  Process  _  _ 

Configuration  Management  Process  _  _ 

Data  Management  Process  _  _ 

Interface  Management  Process  _  _ 

Stakeholder  Requirements  Definition  _  _ 

Process 

Requirements  Analysis  Process  _  _ 

Implementation  Process  _  _ 

Integration  Process  _  _ 

Verification  Process  _  _ 

Validation  Process  _  _ 

Transition  Process 
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System  Engineering  Processes 


Processes 

Decision  Analysis  Process 

Technical  Planning  Process 

Technical  Assessment  Process 

Requirements  Management  Process 

Risk  Management  Process 

Configuration  Management  Process 

Data  Management  Process 

Interface  Management  Process 


Stakeholder  Requirements  Definition 
Process 

Requirements  Analysis  Process 


Description 


Provides  the  basis  for  evaluating  alternatives  and  selecting  the  optimum 
decision. 

Provides  the  critical  quantitative  input  to  program  planning  and  ensures  the 
systems  engineering  processes  are  applied  properly  throughout  the  system's 
life  cycle. 

Measures  technical  progress  and  assesses  both  program  plans  and 
requirements 

Provides  traceability  back  to  user-defined  capabilities  as  documented  through 
either  the  Joint  Capabilities  Integration  and  Development  System  or  other  user- 
defined  source,  and  to  other  sources  of  requirements. 

Overarching  process  that  encompasses  identification,  analysis,  mitigation 
planning,  mitigation  plan  implementation,  and  tracking. 

Establishes  and  controls  product  attributes  and  the  technical  baseline  across 
the  total  system  lifecycle. 

Applies  Policies,  procedures  and  information  technology  to  plan  for,  acquire 
access,  manage,  protect,  and  use  data  of  a  technical  nature  to  support  the  total 
life  cycle  of  the  system. 

Ensures  interface  definition  and  compliance  amount  the  elements  that 
compose  the  system,  as  well  as  with  other  systems  with  which  the  system  or 
system  elements  will  interoperate. 

Elicits  inputs  from  relevant  stakeholders  and  translates  the  inputs  into  technical 
requirements. 

Provides  measureable  and  verifiable  requirements. 


Implementation  Process  Yields  the  lowest  level  system  elements  in  the  system  hierarchy.  The  elements 

that  will  be  combined  together  to  create  the  full  system. 


Integration  Process 

Verification  Process 

Validation  Process 
Transition  Process 


Incorporates  the  lower  level  system  elements  into  a  higher-level  system 
element  in  the  physical  architecture. 

Confirms  that  the  system  element  meets  the  design  to  or  build-to  specifications 
as  defined  in  the  functional,  allocated,  and  product  baselines. 

Answers  the  question  of  "Is  it  the  right  solution  to  the  problem?" 

Moves  any  system  element  to  the  next  level  in  the  physical  architecture. 


*  Reference:  Defense  Acquisition  University.  2011.  Defense  Acquisition  Guidebook. 
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Appendix  C 


Leading  Indicator  Questionnaire 

On  the  attached  page  there  is  a  list  of  18  leading  indicator  trends  and  a  brief  description.  These 
trends  are  used  to  help  give  insight  into  the  direction  a  project  is  going  and  whether  or  not  corrective 
action  needs  to  be  taken.  Please  look  at  each  trend  individually  and  rank  them  1  to  5  on  how  useful  they 
would  be  for  a  sled  test  project,  with  1  being  the  most  useful.  Now  look  at  them  as  a  group  and  rank 
them  1  to  18  with  1  being  the  most  useful  trend  for  a  sled  test  project. 

Trend  Rank  Individually  1-5  Rank  as  a  Group  1-18 

Requirements  _  _ 

System  Definition  _  _ 

Chase  Log 

Interface  _  _ 

Requirements  Validation  _  _ 

Requirements  Verification  _  _ 

Work  Product  Approval  _  _ 

Review  Action  Closure  _  _ 

Risk  Exposure  _  _ 

Risk  Handling  _  _ 

Technology  Maturity  _  _ 

Technical  Measurement  _  _ 

System  Engineering  Staffing  &  Skill  _  _ 

Process  Compliance  _  _ 

Facility  and  Equipment  _  _ 

Availability 

Defect  and  Error  _  _ 

System  Affordability  _  _ 

Architecture  _  _ 

Schedule  and  Cost  Pressure  _  _ 
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Leading  Indicators 


Trend 

Requirements 


System  Definition 
Chase  Backlog 

Interface 


Requirements  Validation 


Requirements  Verification 


Work  Product  Approval 


Review  Action  Closure 


Risk  Exposure 


Risk  Handling 


Description 

Rate  of  maturity  of  the  system  definition  against  the  plan.  Also 
characterizes  stability  and  completeness  of  system  requirements 
which  could  potentially  impact  design  and  production. 

Change  request  backlog  which,  when  excessive,  could  have  adverse 
impact  on  the  technical,  cost  and  schedule  baselines. 

Interface  specification  closure  against  plan.  Lack  of  timely  closure 
could  pose  adverse  impact  to  system  architecture,  design, 
implementation  and/or  V&V  any  of  which  could  pose  technical,  cost 
and  schedule  impact. 

Progress  against  plan  is  assuring  that  the  customer  requirements  are 
valid  and  properly  understood.  Adverse  trends  would  pose  impacts  to 
system  design  and  activity  with  corresponding  impacts  to  technical, 
cost  &  schedule  baseline  and  customer  satisfaction. 

Progress  against  plan  is  verifying  that  the  design  meets  the  specified 
requirements.  Adverse  trends  would  indicate  inadequate  design  and 
rework  that  could  impact  technical,  cost  and  schedule  baselines.  Also, 
potential  adverse  operational  effectiveness  of  the  system. 

Adequacy  of  internal  processes  from  the  work  being  performed  and 
also  the  adequacy  of  the  document  review  process,  both  internal  and 
external  to  the  organization.  High  reject  count  would  suggest  poor 
quality  work  or  a  poor  document  review  process  each  of  which  could 
have  adverse  cost,  schedule  and  customer  satisfaction  impact. 

Responsiveness  of  the  organization  in  closing  post-review  actions. 
Adverse  trends  could  forecast  potential  technical,  costs  and  schedule 
baseline  issues. 

Effectiveness  of  risk  management  process  in  managing  /  mitigating 
technical,  cost  &  schedule  risks.  An  effective  risk  handling  process  will 
lower  risk  exposure  trends. 

Effectiveness  of  SE  organization  in  implementing  risk  mitigation 
activities.  If  the  SE  organization  is  not  retiring  risk  in  a  timely  manner, 
additional  resources  can  be  allocated  before  additional  problems  are 
created. 
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Technology  Maturity 


Technical  Measurement 

Systems  Engineering 

Staffing  &  Skill 

Process  Compliance 

Facility  and  Equipment 
Availability 

Defect  and  Error 

System  Affordability 

Architecture 

Schedule  and  Cost  Pressure 


Risk  associated  with  incorporation  of  new  technology  or  failure  to 
refresh  dated  technology.  Adoption  of  immature  technology  could 
introduce  significant  risk  during  development  while  failure  to  refresh 
dates  technology  could  have  operational  effectiveness  /  customer 
satisfaction  impact. 

Progress  towards  meeting  the  Measures  of  Effectiveness,  Measures  of 
Performance,  Key  Performance  Parameters  and  Technical 
Performance  Measures.  Lack  of  timely  closure  is  an  indicator  of 
performance  deficiencies  in  the  product  design  and  /  or  project  team's 
performance. 

Ability  of  SE  organization  to  execute  total  SE  program.  Includes 
quantity  of  SE  personnel 

assigned,  the  skill  and  seniority  mix  and  the  time  phasing  of  their 
application  throughout  the  program  lifecycle. 

Quality  and  consistency  of  the  project  defined  SE  process.  Poor  / 
inconsistent  SE  processes  and  /  or  failure  to  adhere  to  the  SE  process 
increase  program  risk. 

Availability  of  critical  facilities  and  equipment  needed.  Composed  of 
two  metrics,  one  type  that  measures  facility  availability  and  the  other 
that  measures  equipment  availability. 

Amount  of  defects  and  errors  over  time  for  the  project.  A  defect  is  a 
deviation  of  a  product  at  any  stage  of  its  development, 
implementation,  or  operation  from  its  requirements  or  applicable 
standards. 

The  estimate  of  the  cost  of  the  system  at  the  end  of  the  project  with  a 
given  confidence  and  the  customer's  ability  or  willingness  to  pay  that 
price  for  the  project's  deliverables. 

The  progress  that  an  engineering  team  is  making  towards  developing  a 
comprehensive  system  architecture. 

The  impact  of  schedule  and  cost  challenges  to  the  execution  of  the 
project.  The  percentage  differences  between  project  estimates  and 
contracted  values. 


*  Reference:  Rhodes,  Valerdi,  and  Roedler.  2008.  Systems  Engineering  Leading  Indicators  for  Assessing  Program  and  Technical 
Effectiveness. 

*Reference:  Roedler,  Rhodes,  Schimmoller,  Jones.  2010.  Systems  Engineering  Leading  Indicators  Guide,  Version  2.0. 
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Appendix  D 


Leading  Indicator  Excel  Tool 

To  collect  information  from  the  HHSTT  two  tools  were  created,  one  for  each  of 
the  chosen  LI  trends.  The  tools  utilize  Microsoft  Excel  and  are  written  using  Visual  Basic 
for  Applications  (VBA).  The  user  chooses  what  data  and  trend  lines  they  would  like  to 
track,  they  enter  their  data  and  the  program  computes  a  graphic  containing  the  LI  trend 
information  from  historical  data  and  the  user’s  project’s  data. 

The  tools  are  split  into  four  page  categories:  the  Start  Page,  Project  Input  page, 
Derived  Measures  Input  pages  and  Results  Page.  The  start  page  (Figure  61)  utilizes  the 
LI  measurement  specifications  chart  (Appendix  A)  from  the  ‘Systems  Engineering 
Leading  Indicators  Guide,  Version  2.0’  to  display  the  LI  trend  characteristics.  This  chart 
is  updated  with  user  inputs  from  the  Project  Input  page.  From  the  Start  page  the  user  has 
the  options  to  start  a  new  project,  open  the  exiting  project  or  go  directly  to  the  results 
page.  The  new  project  button  will  erase  all  prior  user  inputs  and  bring  up  the  Project 
Input  page.  The  exiting  project  button  will  keep  all  prior  user  inputs  and  bring  up  the 
Project  Input  page.  The  Graph  button  will  bring  up  the  results  page.  The  program  is 
designed  to  always  open  to  this  page. 
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Figure  61:  LI  Tool  Start  Page 
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The  Project  Input  pages  (figure  62  &  63)  are  where  the  user  chooses  what  they 
would  like  to  track  and  how  often.  The  first  input  is  who  the  Project  Manger  (PM)  is  for 
this  project.  The  second  input  is  what  is  being  tracked.  Each  LI  trend  is  broken  down 
into  their  derived  measures.  The  requirements  trend  has  five  derived  measures  to  choose 
from;  percent  of  requirements  approved,  percent  of  requirements  growth,  percent  of 
requirements  TBD/TBR,  percent  of  requirements  modified  and  estimated  affected  hours. 
The  requirements  validation  trend  has  two  derived  measures  to  choose  from; 
requirements  validation  rate  and  percent  requirements  validated.  The  next  two  input 
sections  are  data  collection  and  data  reporting.  The  user  enters  the  frequency  of  the  data 
collection/reporting  in  weeks,  the  start  data  of  the  collection/reporting  and  the  estimated 
number  of  collections/reports  for  the  project.  The  program  calculates  the  end  date.  The 
number  of  collections/reports  can  be  updated  at  any  point  in  the  project,  in  turn  updating 
the  end  date,  to  accommodate  changes  in  the  length  of  the  project.  A  typical  project’s 
data  collection  and  data  reporting  will  be  the  same;  however  for  the  rare  occasion  when 
they  are  different  the  tool  lists  them  separately.  The  inputs  from  this  page  are  transferred 
to  the  LI  measurement  specification  chart  on  the  Start  page.  The  next  button  formats 
each  of  the  Derived  Measures  Input  pages  and  brings  up  the  first  Derived  Measures  Input 
page  that  has  been  selected.  Any  derived  measure  that  had  not  been  selected  will  not  be 
displayed  while  transitioning  through  the  tool. 
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Figure  62:  Requirements  Trend  Project  Input  Page 


Figure  63:  Requirements  Validation  Trend  Project  Input  Page 

The  Derived  Measures  Input  pages  are  configured  for  the  user  to  input  the 

required  data  next  to  their  respected  date.  The  dates  are  preset  on  the  page  from  the  data 

imputed  on  the  Project  Input  page.  The  tool  completes  any  calculations  needed 
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simplifying  the  input  process  for  the  user.  The  Next  button  brings  up  the  next  Derived 
Measures  Input  page.  When  all  Derived  Measures  Input  pages  have  been  viewed  the  last 
Next  button  brings  up  the  Results  page. 

Within  the  requirements  trend  tool  the  Percent  of  Requirements  Approved  Input 
page  (Figure  64)  allows  the  user  to  enter  the  number  of  known  requirements  and  the 
number  of  requirements  that  have  been  approved.  The  Percent  of  Requirements  Growth 
Input  page  (Figure  65)  allows  the  user  to  enter  the  number  of  new  requirements.  The  tool 
calculates  the  number  of  old  requirements  and  the  total  number  of  requirements  from 
previous  inputs.  The  Percent  of  Requirements  TBD/TBR  Input  page  (Figure  66)  allows 
the  user  to  enter  the  number  of  requirements  TBD/TBR  open  and  the  number  of 
requirements  TBD/TBR  closed.  The  tool  calculates  the  total  number  of  requirements  that 
are  or  have  been  TBD/TBR.  The  Percent  of  Requirements  Modified  Input  page  (Figure 
67)  allows  the  user  to  enter  the  total  number  of  requirements  and  the  number  of  the 
requirements  modified.  The  Estimated  Affected  Hours  Input  page  (Figure  68)  allows  the 
user  to  enter  the  additional  hours  needed  to  complete  the  project  due  to  requirements 
changes  or  late  requirements  for  that  collection  period.  The  tool  calculates  the  previous 
additional  work  hours  and  the  total  number  of  additional  work  hours  needed  to  complete 
the  project  to  date. 
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I  Percent  of  Requriements  Approved 
1 1)  Enterthe  number  of  requirements  known 
I  2)  Enterthe  number  of  requirements  that  have  been  approved 
1 3)  Press  the  yellow  Next  button 


Next 


Date 

Number  of 

Customer 

Requirements 

Known 

Number  of 

Customer 

Requirements 

Approved 

Number  of 

Derived 

Requirements 

Known 

Number  of 

Derived 

Requirements 

i-nov  in 

3 

2 

2 

0 

15-Nov-I0 

5 

2 

2 

0 

29  Nov  10 

5 

2 

2 

0 

13- Dec-10 

5 

2 

2 

0 

27-Dec-10 

5 

4 

2 

1 

in  J.»»  11 

24-Jwi-ll 

7-Feb-ll 

21-Feb-ll 

7-Mw-ll 

21-Mw-ll 

4-Apr-ll 

lS-Apr-11 

2- May- 11 

16- May-11 

30- May-11 

13-Jiai-ll 

27-Jim-ll 

11-JiJ-ll 

25-JJ-ll 

RAuB  11 

22  Aug  11 

5-Sep-ll 

19-Sep-ll 

3-Otl-ll 

17-Od-ll 

Figure  64:  Percent  of  Requirements  Approved  Input  Page 


Figure  65:  Percent  of  Requirements  Growth  Input  Page 
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I  Percent  of  Requirement  TBD/TBR 

llj  Enterthe  number  of  TBD/TBR  requirements  open 

12)  Enterthe  numberof  TBD/TBR  requirements  dosed 

1 3)  Press  the  yellow  Next  button 


Next 


Date 

Numberof 

TBD/TBR 

Customer 

Requirements 

Open 

Number  of 
TBD/TBR 
Customer 

Requirements 

Closed 

Total  Number 
of TBD/TBR 
Customer 

Requirements 

Number  of 
TBD/TBR 
Derived 

Requirements 

Open 

Number  of 
TBD/TBR 
Derived 

Requirements 

Closed 

Total  Number 
of TBD/TBR 

Derived 

Requirements 

1-Ncw-lD 

2 

3 

5 

3 

0 

3 

0 

5 

5 

3 

0 

3 

|  29Uo*  ID 

0 

5 

5 

3 

0 

3 

0 

5 

5 

3 

0 

3 

27-Dec- ID 

0 

5 

5 

2 

1 

3 

It)  Inn  11 

I  7-Feb-U 

7MrU 

21-Mar- 11 

4-A|jrU 

IK  A|M  11 

2-Htay-ll 

IfrMay-U 

13-Jim-ll 

27-Jun-ll 

11  hi  11 

SJdU 

8- Aug-11 

If  A.^  11 

5-Sep-ll 

19-5ep-ll 

3- Oct-11 

17-Octll 

Figure  66:  Percent  of  Requirements  TBD/TBR  Input  Page 


I  Percent  of  Modified  Requirements 

1 1)  Enterthe  total  numberof  requirements 

1 2)  Enterthe  number  requirements  modified 

1 3)  Press  the  yellow  Next  button 


Date 

Number  of 

Customer 

Requirements 

Number  of 

Modified 

Customer 

Requirements 

Number  of 

Derived 

Requirements 

Number  of 

Modified 

Derived 

Requirements 

1-Nov-lD 

5 

3 

3 

0 

5 

2 

3 

1 

29- Nov-10 

5 

2 

3 

2 

13- Dec- ID 

5 

1 

3 

2 

27-Dee -ID 

5 

0 

3 

1 

ID- Jan- 11 

24- Jan- 11 

7- Feb- 11 

21- Feb- 11 

7-Mar- 11 

21- Mar-11 

4-Apr-ll 

lfJAprll 

2-May- 11 

1&  May-11 

3D- May- 11 

13-Jun-ll 

27-Jun-ll 

HJiJll 

25-JJ-ll 

8-Aitg-ll 

22- Aug-11 

'>  Sep  11 

19- Sep- 11 

loan 

17-Oct-ll 

Figure  67:  Percent  of  Requirements  Modified  Input  Page 
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Figure  68:  Estimated  Affected  Hours  Input  Page 


Within  the  requirements  validation  tool  the  Requirements  Validation  Rate  page 
allows  the  user  to  list  all  the  project  requirements  with  the  date  the  requirement  was 
created  and  the  date  the  requirement  was  validated  (Figure  69).  A  drop  down  menu 
allows  the  user  to  choose  the  type  of  requirement,  C  for  customer  or  D  for  derived.  The 
Percent  of  Requirements  Verified  page  allows  the  user  to  enter  the  total  number  of 
requirements  and  the  total  number  of  requirements  that  have  been  verified  (Figure  70). 
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I  Requirements  Validation  Rate 
1 1)  Enter  the  requirement  to  be  validated  (upto  200) 

I  2)  Enter  the  date  the  requriment  to  be  verified  was  created 

1 3)  Enter  the  date  the  requirment  was  validated 

1 4)  Using  the  drop  down  menu  chose  the  type  of  requirment 
C- customer  D- derived 

I  5)  Press  the  yellow  Next  button 


Requirement  to  be  Verified 

Date 

Requirement 
was  Created 

Date 

Requirement 
was  Validated 

Requirement 

Type 

IMPACT  VETO  OIY 

10-Nov-JD 

C 

IMPACT  ATTTTUDE 

1  Nov  ID 

1  NovlO 

C 

TARGET  SZE 

1  Nov  in 

1  Nov  10 

c 

TARGET  HARDNESS 

i  Nov  in 

ISDec-JO 

c 

TARGET  ARRANGEMENT 

10  Nov  in 

ISOec-JO 

c 

FOREBODV  STED 

1-Nov-lQ 

IS-Dec-JD 

D 

PUSHER  SLEDS 

1-Nov-lQ 

D 

PHOTO  PLAN 

INovIO 

D 

Figure  69:  Requirements  Validation  Rate  Input  Page 


Figure  70:  Percent  of  Requirements  Validated  Input  Page 

The  Results  page  shows  the  tools  output  in  form  of  graphs  (figure  71  &  72).  Each 
derived  measure  has  a  separate  graph.  The  graphs  for  the  derived  measures  that  were  not 
chosen  will  be  blank.  All  of  the  graphs  are  line  graphs  except  for  the  requirements 
validation  rate  derived  measure  which  is  a  bar  graph.  The  graphs  contain  a  historical 
trend  line  and  a  project  trend  line.  The  tool  scales  the  historical  trend  line  to  match  the 
length  of  the  chosen  project.  They  also  contain  a  vertical  line  depicting  the  current  date. 
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The  user  uses  these  graphs  and  their  knowledge  of  the  project  and  organization  to  make 
informed  decisions  about  the  project  and  whether  or  not  the  corrected  actions  need  to  be 
made.  The  Save  Project  button  opens  the  save  as  function,  the  Edit  Project  button  takes 
the  user  to  the  Inputs  page  and  the  Start  Page  button  takes  the  user  to  the  Start  page. 


Figure  71:  Requirements  Trend  Results  Page 


Figure  72:  Requirements  Validation  Trend  Results  Page 
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This  tool  was  designed  for  high  speed  impact  tests  conducted  at  the  Holloman 
High  Speed  Test  Track.  For  this  tool  to  be  utilized  by  any  other  type  of  project  or  by  any 
other  organization  the  historical  data  imbedded  in  the  tool  would  need  to  be  modified. 
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